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ABSTRACT 


During the period of 20-25 October 2008, a training research exercise (TREX) 
was conducted to integrate operational concepts and training techniques from different 
commands. The collaborative teamwork demonstrated in the highly asymmetric threat 
exercise scenario was recorded in Microsoft internet relay chat logs across fifteen 
different chat rooms. The goal of this thesis was to use chat room recorded data from the 
Air Operations Center to evaluate a measurement model of macrocognition developed 
under the Collaboration and Knowledge Integration program, sponsored by the Office of 
Naval Research. The model focuses on team member’s cognitive processes used during 
collaboration with the goal of trying to understand how individuals collaborate to build 
new knowledge and accomplish their tasks. Effective chat communications may expedite 
the process of moving the team towards achieving their ultimate goal which was to 
produce optimum combat effectiveness in a timely manner. Thesis results will be 
provided to the Office of Naval Research to help improve collaboration among teams 
while operating in stressful and dynamic environments. 
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I. 


INTRODUCTION 


A. THESIS GOAL 

1. Collaboration and Knowledge Interoperability 

The Department of Defense is being confronted by a highly capable and quickly 
adaptable adversary. The adversary fights in a constantly changing and dynamic 
environment using asymmetrical warfare techniques. In order to effectively combat and 
defeat such an enemy, human decision makers at the tactical, operational and strategic 
levels must act and decide rapidly. To enable personnel to make quicker and more 
accurate decisions, personnel at all levels of the chain of command must be given access 
to more detailed information than ever before and collaborate with several inter-theater 
and cross border multinational and multiagency personnel. 

The Office of Naval Research (ONR) sponsored Collaboration and Knowledge 
Interoperability program attempts to gain insight and understanding into the human 
cognitive processes demonstrated during an individual’s (or group of individuals) attempt 
at solving extremely complex, time sensitive, and dynamic problems. ONR studies 
suggest that by gaining greater insight and understanding of all the metacognitive and 
macrocognitive processes exhibited by humans during the entire decision making 
process, better system architectures, models and methods can be developed to improve 
the reliability and speed of data dissemination to the human decision maker. 
Additionally, creation of human agent interfaces based on human factor principles would 
provide better information displays to increase user understanding and ease of system use 
therefore improving effectiveness of decision making (Letsky, 2008). 

ONR developed a model of team collaboration and cognitive processes (Figure 1) 
that includes the input factors (time pressure, information uncertainty, dynamic 
environment factors, etc...), and phases (knowledge construction, collaborative team 
problem solving, team consensus, and outcome evaluation and revision) demonstrated 
during team collaboration. 
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Mechanisms for achieving Meta and MacroCognitive Processes fapplies to all stages) 

* representing and discussing individual information, discussing team generated information, 
questioning, agreeing / disagreeing, negotiating perspectives, discussing possible solutions, providing rationale. 

• \nn-\prh al commi i nic a t i nn s; facial expressions, voice clues (vocal paralanguagc), hand gestures, body movcmcnts(kinc^i 
touch (lapiic^, personal space, drawing, text messages, augmented video, afTordanccs(cognition in objects). 


Figure 1. Collaboration Stages and Cognitive Process Model (From Warner, Letsky, & 

Cowan, 2005). 


The Model of Team Collaboration, shown in Figure 1, allows researchers the 
ability to try and predict human decision-making outcomes in specific real-world events 
and exercise scenarios (Warner, Letsky, & Cowan, 2005). 

2. Goals for the SUMMIT MURI Research 

In October 2008, an evolutionary change to the original Model of Team 
Collaboration was suggested during the Office of Naval Research Systems for 
Understanding and Measuring Macrocognition in Teams (SUMMIT) meeting sponsored 
by a Multidisciplinary University Research Initiative (MURI) grant. The evolutionary 
changes were needed to address human related collaboration issues that have been 
discovered as the Department of Defense steadily increases its reliance and dependence 
on network centric warfare. Network centric warfare requires strong communication and 
detailed coordination among all operational and tactical units located within or outside 
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the operational theater. For network centric warfare to be successful several 
geographically separated operational and tactical personnel must be able to quickly form 
into a cohesive team. Despite possible individual biases, the team must be able to 
effectively share and process information (convert data to information and information to 
knowledge) in order to facilitate faster decision making and effectively combat the 
enemy. The MURI team conducted detailed research and investigated how the delivery, 
availability and different types and information and information processing systems 
affected the macrocognitive processes within teams that were both centrally located and 
distributed. The MURI research goal is to improve collaboration process understanding 
and creation of better tools to better support macrocognition in teams. Additionally, the 
research focused on creating a test environment and produced network centric warfare 
cognitive metrics that can be used to evaluate team cognitive behavior (Fiore, Rosen, 
Salas, Burke, Warner, & Letsky, in press). 

3. Major Goals for this Thesis 

The major goal of this thesis is to evaluate the new Office of Naval Research 
sponsored Measurement Model for Macrocognition Research using the new SUMMIT 
coding definitions, dated October 2008. Definitions of the macrocognitive processes 
included in the model were applied to the Training Exercise (TREX) 2009 Air Operations 
Center Dynamic Targeting Cell generated data. Eigure 2 is a graphical representation of 
the measurement model metacognitive phases and illustrates how individuals functioning 
as a team process and learn new information. The new SUMMIT coding definitions will 
be discussed in detail in Chapter IV of this thesis and the coding process will be 
explained in Chapter V of this thesis. 
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Legend: 



Figure 2. Measurement Model for Macrocognition Research (After Fiore, et ah, in 

press). 


The TREX 09-1 exercise conducted during the period of 20-22 October 2008 
provides us with a realistic, dynamic, time-sensitive situation to asses and validate the 
measurement model. By coding the data from the exercise utilizing the new SUMMIT 
definitions, we can determine if the metacognitive processes shown in the measurement 
model (Figure 2) are an actual representation of how teams collaborate to handle dynamic 
problems in a time sensitive environment. 

During coding it was discovered that there were existing gaps in the measurement 
model and the SUMMIT definitions. Theses gaps are explained in Chapter VI, results 
section of this thesis, along with recommendations to the definitions. Additionally, some 
of the SUMMIT definitions were not used during our coding of the TREX 09-1 data. A 
brief explanation of why certain codes weren’t used is included in Chapter VI. 
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Goals for the AOC Dynamic Effects/Targeting Cell 


The main goals of TREX 09-1 are listed in Table 1. For the purpose of this thesis 
the terms Dynamic Effects Cell (DEC) and Dynamic Targeting Cell (DTC) are 
synonymous. 

Table 1. Training Research Exercise 2009-1 Dynamic Effects Cell Research 

Objectives (From Armed Forces Research Eaboratory Warfare Readiness Division, 

2008). 


TREX 09-1 Dynamic Effects Cell Research Objectives 


Goal #1: Assess effects of chat protocol on communications and mission performance: 

- Measure effectiveness of chat room user 

- Measure of efficient times from info drop to engagement 
Goal #2: Highlight and explore future concepts for continuous learning: 

- Use individual training sessions to train in Air Operations Center environment. 

- Assess war fighter planning, employment and effectiveness 

Goal #3: Integrate war fighter controllers and information simulation exercise 
participants: 

- Build and employ Master Scenario Event Eist and Request Eor Information 
management environment. 

- Engage Senior Intelligence Duty Officer and Target Duty Officer with realistic 
interactions and survey for effect 

Goal #4: Use of multiple performance measurement systems 

Goal #5: Assess effects of lED Network Defeat Tactics, Techniques and Procedures for 
non-kinetic mission execution 


This thesis evaluated the team collaboration within the Air Operation Center 
(AOC) DEC which is a contributing factor for measuring the effectiveness of all the 
TREX 09-1 research goals. Feedback and the results of this thesis will be provided to the 
Air Force Research Eaboratory War-fighter Readiness Research Division to allow them 
to assess the data and possibly assist them in developing more realistic AOC DEC 
exercise scenarios. 
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B. AIR OPERATIONS CENTER DYNAMIC EFFECTS/TARGETING CELL 


I. Mission 

After five years in Afghanistan and Iraq, the United States and Coalition forces 
remain actively engaged against a highly capable and adaptive enemy. Unlike the Cold 
War standoff with the Soviet Union that matched nation against nation, massive force 
against massive force, and military technology against military technology, today’s 
enemy does not fight for a particular country, has no visible army to call its own, and has 
very limited technology, capabilities and resources to combat coalition forces. The 
enemy does however have a very strong will to fight and the means by which to use 
irregular warfare techniques to create mass fear and to disrupt the life of those friendly to 
the United States’ way of life. The enemy’s employment of improvised explosive 
devices (lEDs) has caused massive fear, severe damage, and the loss of life of several 
sailors, soldiers, and countless Iraqi civilian personnel. To counter the lED threat and 
take advantage of the limited targeting window provided by those responsible for lED 
employment, new ways have been adapted by the AOC to quickly and accurately conduct 
dynamic targeting against the enemy and lED threats. 

Adaptation of a dynamic effects cell within the AOC (Joint and Combined) 
command structure, Eigure 3, provides the Joint Eorce Air Component Commander 
(JEACC) the leverage to engage and attack targets of opportunity that meet Joint Eorce 
Commanders mission objectives. Dynamic targeting is a means by which coalition 
forces can respond to the employment of lEDs and insurgent network leadership located 
within Afghanistan and Iraq. Due to the short time-sensitive nature, dynamic targets are 
normally vetted quickly and accurately through the targeting cycle. This process allows 
the JEACC to task available assets or reassign assets to engage and destroy known targets 
or other potential threats at a moment’s notice. The limited time window available for 
target engagement and destruction means that coalition forces must be prepared to apply 
timely and accurate measures and counter-measures against the enemy. 
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CPOF/ABCS data link 
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Figure 3. Dynamic Effects Cell in the Command Structure (From U.S. Air Force 

Central Command, no date). 

2. Dynamic Effects Cell Area of Responsibilities 

The DEC is responsible for directing the planning and coordination of all dynamic 
targets, time-sensitive targets (TSTs), and high payoff target operations (U.S. Air Force 
Central Command, no date). Highly trained personnel and the advances in intelligence, 
surveillance and reconnaissance (ISR) assets provide the necessary tools to find, fix, 
target, track, engage, and assess targets of opportunity that require immediate attention 
and/or action. The airborne warning and control system, joint surveillance targeting 
attack radar system and unmanned aerial vehicles continue to provide the Commander of 
U.S Central Command and the Combined Force Air Component Commander (CFACC) 
the means to destroy TSTs such as lEDs, key insurgent leadership personnel, 
and terrorist network cells. Of the 3500 targets nominated by the CFACC during 
Operation Iraqi Freedom, 156 were classified as TSTs and 686 were identified as 
dynamic targets (Moseley, 2003). 
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3. 


Typical Operational Components and Players 


The DEC falls within the Combat Operations Division of the AOC. Dynamic 
target nomination is coordinated with the combat operations division via the Senior 
Offensive Duty Officer (SODO). The organization makeup depends on the overall 
operational requirements and Joint Force Commander (JFC) mission objectives. If 
included as a component of the AOC, the DEC at a minimum will include a DEC Chief, a 
Deputy DEC Chief, a Ground Track Coordinator, an Attack Coordinator, and a Target 
Duty Office as shown in Figure 4 (Secretary of Air Force, 2005). Proper synchronization 
of ISR collection support for target tracking and engagement between the Senior 
Intelligence Duty Officer (SIDO) and DEC is vital for overall AOC mission success and 
for de-confliction of friendly fires. 
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Figure 4. Dynamic Effects Cell Organizational Structure (From U.S. Air Force Central 

Command, in press). 


4. AOC Organization and Responsibilities 

Today’s AOC is the focal point from which planning, directing, controlling and 
coordinating of air and space operations comes together to satisfy the JFC objectives. 
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The AOC must be prepared to carry out deliberate dynamic targeting without 
compromising its capability to conduct other major air combat operations (U.S. Air Force 
Central Command, no date). According to the Air Force Instruction 13-1 AOC Volume 
3, a typical AOC command structure (Figure 3) will consist of an AOC Director, five 
divisions (Strategy, Combat Plans, Combat Operations, Intelligence Surveillance and 
Reconnaissance, and Air Mobility), and multiple support specialty teams. Although not 
all inclusive, such a command structure is extremely vital to AOC mission success. 

5. Joint Force Air Component Commander Responsibilities 

The JFACC is assigned by the JFC to manage the AOC. The JFACC is 
responsible for effective planning, coordination, allocation, and theater tasking of assets 
to accomplish the JFC mission (Joint Publication 3-30, 2003). The JFACC accomplishes 
the assigned mission in accordance with the guidance and under the authority granted by 
the JFC. The JFACC exercises operational and tactical control over all assigned 
personnel and assets located within the theater of operation. The allocation of assets to 
destroy key enemy leadership, suppression of enemy air defense systems, convoy 
protection, close air support, and dynamic targeting are just a few of many ways by 
which the JFACC could use the assigned assets to render an adversary’s method of attack 
ineffective and thereby minimize damage and coalition force and civilian casualties. 

C. TRAINING EXERCISE 2009 

During the period of 20-25 October 2008, an intense training research exercise 
(TREX 09-1) was conducted in Mesa, Arizona, bringing together several key operational 
command military and civilian personnel from all services. TREX 09-1 brought together 
personnel from the United States Air Eorce Warfare Center, Naval Strike Air Warfare 
Center, Special Operations Command, and United States Army (Armed Eorces Research 
Eaboratory Warfare Readiness Division, 2008). The purpose of this TREX 09-1 exercise 
was to assess tactics, techniques and procedures of operational command personnel 
performing kinetic and non-kinetic dynamic targeting in a highly asymmetric 
environment. 
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The TREX 09-1 exercise was a simulation of a 12-hour overnight shift in the 
dynamic effect cells of a typical AOC. The 12-hour shift was broken into six 2-hour time 
sections. The operational players were separated during the exercise by use of false walls 
(i.e., room dividers) to simulate real-world space and time disparity of the various joint 
service personnel. Figure 5 shows seating chart layout of exercise personnel. 
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Figure 5. Air Operations Center Testbed and Intelligence Surveillance and 
Reconnaissance Seating Chart (From Armed Forces Research Faboratory Warfare 

Readiness Division, 2008) 


Communication between the various players was facilitated by the use of the Joint 

Automated Deep Operations Coordination System. Joint Automated Deep Operations 

Coordination System is a command and control software application that provides 

various tools for all users to see the same tactical picture and to facilitate collaboration 

and integration across the battle space environment (Jane’s, 2009). TRFX 09-1 exercise 

communications were recorded across fourteen different chat room channels. Figure 6 

identifies the chat rooms and specific personnel inside each chat room. 
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Figure 6. Joint Automated Deep Operations Coordination System Chat Room Layout 
(From Armed Forces Research Laboratory Warfare Readiness Division, 2008). 


Each of the TREX 09-1 operational players had varied access and responsibilities 
in the chat rooms used for the exercise. All the operational players did not have access to 
every individual chat room. Some key operational players were designated as room 
owner (“R”), active participant (“P”) and/or observer (“O”). Figure 7 is a list of 
participants and their associated responsibilities in each chat room. 
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Figure 7. Joint Automated Deep Operations Coordination System Chat Room 
Assignments (From Armed Forces Research Laboratory Warfare Readiness 

Division, 2008). 
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II. BACKGROUND 


Chapter II Background Section A of this thesis is intended to bring the reader up- 
to-date on the history and types of missions the Air Operations Center (AOC) supports in 
the battle space. Section B will advise the reader on the internal structure of the AOC 
and responsibilities of key personnel. Section C will inform the reader on how the AOC 
Dynamic Targeting Cell (DTC) is designed to handle asymmetric threats and describe 
specifically how DTCs respond and prosecute time sensitive targets. 

A. AIR AND SPACE OPERATIONS CENTER 

I. History behind AOC Conceptualization 

The 11 September 2001 attacks on the World Trade Center and the Pentagon 
revealed that U.S. homeland security was extremely vulnerable and lacked the procedures 
for an effective coordinated response to combat such highly coordinated and dynamic 
attacks. At the time of the 9/11 attacks, North American Aerospace Defense (NORAD) 
command, a joint endeavor between U.S. and Canada, was responsible for air security 
and air sovereignty defense for all of North America. NORAD’s primary mission at the 
time of the terrorist attacks only considered the Soviet Union as a major threat and 
therefore NORAD only trained and planned to combat and respond to over-the-horizon 
strategic-strike ballistic missile attacks form the Soviet Union. An attack from nineteen 
terrorists using U.S. hijacked airplanes was never anticipated or planned for which caught 
NORAD unprepared to respond appropriately. Air Force General Myers, Chairman of 
the Joint Chiefs of Staff, later defended NORAD’s poor response to the 9/11 attacks by 
stating that NORAD was not tasked or responsible for attacks originating from with 
inside the U.S. but only responsible for those attacks that originated from outside the U.S 
(Banusiewicz, 2004). 

To deter aggression and detect threats to the defense of the U.S. and our allied 
nations, AOCs are being built across the globe. AOCs with their clearly defined roles 
and responsibilities may shorten response times and eliminate the mass confusion that 
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was experienced by NORAD during the 9/11 attacks. Linking the AOCs will allow 
operators to share real-time information between AOC centers located in the U.S. and 
those positioned abroad. According to David Fulghum, a Senior Military Editor for 
Aviation Week & Space Technology, “a common or single, integrated air picture would 
let operational commanders or intelligence analysts, for example, follow the flight of a 
suspicious aircraft as it moves across international borders or from one theater to 
another.” (Fulghum, 2004, 58). It is expected that integration of AOCs will result in 
reduced reaction times between AOC centers and provide seamless pass off tasking from 
one AOC to another when disaster strikes. 

2. AOC Types and Responsibilities 


A total of 23 AOC sites have been established on air force bases (AFB) located in 
the Continental U.S. and in several other countries around the world. Figure 8 shows the 
locations of all twenty-three AOC sites. 



Figure 8. Air Operations Center Focation Map. (From Murray, 2008). 
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AOCs are divided into four different groups according to the type of operational 
support they provide to the Joint Force Air Component Commander (JFACC). The four 
groups are Falconer, Tailored, Functional and Support. All AOCs, on short notice, can 
provide the JFACC with the appropriate air assets and trained personnel needed to 
respond to threats originating from both inside and outside the continental U.S. Falconer 
AOCs are responsible for directly supporting their assigned theater regional commander. 
Tailored AOCs support missions ranging from homeland security to strategic defense. 
Functional AOCs are tasked with supporting the U.S. Strategic Command and the U.S. 
Transportation Command. The other support AOCs across the U.S. undertake roles 
dealing with training personnel along with various other support related functions. 

The push to ensure Commanders have the greatest and newest technology 
available to them in order to protect the U.S. homeland and our way of life is evident in 
the integration of AOCs across the country. In May 2007, the dual-hated Commander of 
the 12th AF and AFSOUTH opened the first and only U.S. “Falconer” AOC at the time, 
at Davis Monthan AFB in Arizona. The new AOC center is responsible for supporting 
air and space missions in both Central and South America and in the Caribbean (Jackson 
& Broshear, 2007). The 1st AF, home based at Tyndall AFB Florida, also opened a new 
Tailored AOC in June 2007. As a member of the Air Component Command, the 1st AF 
directs and controls all activities for NORAD within the continental U.S. . 

3. AOCs Designated as a Weapons System 

The AOC was designed as a weapon system by the Chief of Staff of the AF, 
General Michael E. Ryan, in September 2000. While visiting Hurlburt Field, Florida 
aerospace operations center. General Ryan had the following to say: 

I declare the AOC as an official weapons system today. During a real- 
world operation, the AOC will be the "eyes, ears, hands and legs of the 
commander. In each of our theaters, the ability of the air commander to 
execute the missions he has depends on the capability to have an 
aerospace operations center that (can be tailored) ... for the mission he 
needs to do. We need a base lining of the capabilities in that weapons 
system, just as we do in our capabilities in something like an F-16. (In the 
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F-16), we have a crew chief that knows how to maintain it and we have 
pilots that know how to fly it. We have to have the same concept for our 
aerospace operations centers (U.S. Department of Air Force, 2000, 1). 

According to the Air Force, the AOC was designated as a weapon system to 
ensure standardization in the way centers are equipped, employed, and trained (Sirak, 
2006). This standardization across AOCs is expected to (1) produce seamless 
coordination between operators, (2) reduce cost for equipment, (3) decrease compatibility 
issues between equipment, and (4) provide better personnel training standards. 

B. INTERNAL STRUCTURE OF AOC AND RESPONSIBILITIES 
I. Internal AOC Organizational Structure 

To be operationally effective, all division leaders and internal office personnel 
must stay abreast of the common operating picture and coordinate with other division 
personnel in a timely manner. Coordination and collaboration are essential to being able 
to quickly assess the dynamic situation, develop effective courses of action, and assign 
appropriate assets to respond to dynamic events. There are several internal divisions 
within an AOC with each division having specific responsibilities and an assigned leader 
for coordination. Figure 9 shows the internal AOC organizational divisional structure. 
During the TREX 09-1, the Senior Operations Duty Officer (SODO), the Senior Air 
Defense Officer (SADO), the Senior Intelligence Duty Officer (SIDO) and other analysts 
and key specialty/support personnel (i.e.. Marine Liaison, Navy Liaison, Battlefield 
Coordination Detachment) were actively engaged in the coordination of assets and 
capabilities. Additionally, the above personnel relayed and inputted critical data and 
information into the team macrocognition process. 
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Figure 9. Air Operations Center Internal Organization Structure (From Air Force 
Tactics, Techniques, and Procedures 2-3.2, 2004). 


2. Offensive Division and SODO Responsibilities 


The Offensive Division is led by the Senior Operations Duty Officer (SODO) and 
contains several offensive duty officers and the DTC. The SODO and offensive duty 
officers are responsible to the Chief of Combat Operations for ensuring that air tasking 
orders operations meet the JFACC objectives. On-going collaboration with the Wing 
Operations Center and other service control agencies using the contingency theater 
automated planning system is necessary to ensure air assets are used effectively to meet 
objectives. The SODO’s specific coordination actions supporting the TREX 09-1 
included analyzing air tasking requests to ensure all tasks were attainable; tracking all air 
tasking order missions and developing of alternate courses of action in case of emergency 
or change in the common operating picture; and ensuring all changes and modifications 
to the air tasking order were coordinated with appropriate units and disseminated to all 
units affected/concemed (12* Air Force Air Forces Force, 1996). 
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3. 


Defensive Division and SADO Responsibilities 


The Defensive Division is controlled by the Senior Air Defense Officer (SADO) 
and contains several defense duty officers, theater missile defense, and the interface 
control officer. The SADO and defense duty officer are responsible to the chief of 
combat operations for all air defense operations. During TREX 09-1, the SADO and 
defense duty officer responsibilities included joint planning, coordination, employment 
and life-cycle management of all air defense systems in the theater. Additionally, they 
assisted in developing courses of action for countering enemy offensive air activities and 
served as the airspace control authority developing standard operating procedures for air 
battle operations. Effective collaboration between the SADO, defensive duty officers, 
theater missile defense and interface control officer is considered essential to ensure 
protection of U.S. and ally forces and civilians located within the area of responsibility 
(Joint Warfare Publication, 2003). 

4. ISR Division and SIDO Responsibilities 

The Intelligence, Surveillance, Reconnaissance Division (ISRD) is managed by 
the Senior Intelligence Duty Officer (SIDO) and contains the ISR cell which employs 
several collection managers, analysts, and targeteers. During the TREX 09-1, the ISRD 
was responsible for developing the overall ISR strategy for the AOC and planning and 
executing ISR missions. Additionally, the ISRD performed detailed assessments and in- 
depth analysis of the battle space environment and disseminated the information to other 
TREX 09-1 participants. (United States Air Eorce Central Command, 2008). Eigure 10 
shows a more detailed list of ISRD assessment and planning functions that must take 
place in every AOC or training scenario to enable the different internal cells to complete 
their assigned missions. 
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Figure 10. ISRD Assessment and Planning (From United States Air Force Central 

Command, 2008). 


5. Specialty/Support Division Responsibilities 

Several key personnel in the specialty/support division provide necessary support 
to AOC internal cells performing live or training missions, such as the JAG Officer, 
Battlefield Coordination Detachment, Naval and Amphibious Liaison Element (NALE), 
Marine Liaison Officer (MARLO), and the Special Operations Liaison Element (SOLE). 
The JAG Officer provides legal guidance to the internal AOC divisions and operational 
commanders located in the area of responsibility to assist with rules of engagement 
determination. The battlefield coordination detachment is typically an Army assigned 
liaison officer who is responsible for handling and processing tactical air support and de- 
conflicting air operations for the Army component in theater. During the TREX 09-1, the 
battlefield coordination detachment assisted in planning and coordinating air operations 
by interpreting and introducing Army ground warfare techniques (Joint Publication 1-02, 
2001). The NALE and MARLO liaisons also supported the TREX by planning and 
coordinating naval and Marine Corp specific capabilities such as air support, ISR 
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capabilities, and amphibious assault operations. The NALE counsels the AOC internal 
division planners of the maritime domain awareness picture and the MARLO advises the 
AOC internal division on marine ground specific operations and techniques (12* Air 
Force Air Forces Force, 1996). 

C. COLLABORATION BETWEEN OPERATIONAL PLAYERS 

1. Time Sensitive Target Contributing Factors and Process 

The AOC is tasked with carrying out both static and dynamic operations. Static 
operations include maintaining the theater common operating picture, developing and 
analyzing ISR data to maintain accurate situation awareness, supplying weather 
forecasting services, and providing Judge Advocate General legal services for rules of 
engagement determination. Dynamic operations can include finding and destroying 
SCUD missile sites; locating, tracking and destroying radiological or explosively formed 
projectiles logistic networks; and locating, tracking, and/or exploiting key leadership 
targets (Armed Forces Research Faboratory Warfare Readiness Division, 2008). The 
method employed by AOCs, depicted in Figure 11, is the Find, Fix, Track, Target, 
Engage, and Access (F2T2EA) process which enables better command and control 
decision flow and sensor coordination (Joint Publication 3-60, 2007). 

The complexity inherent in performing the tasks involved in static operations 
coupled with a constantly changing environment of dynamic mission factors (i.e., number 
of targets, type of targets, and speed of targets) present challenging problems. These 
extremely challenging problems can be difficult to assess, orchestrate measures and 
countermeasures, and implement courses of action successfully without having effective 
and efficient coordination and de-confliction within the AOC internal organization 
divisions. Collaboration between the internal divisions is paramount to maintaining an 
up-to-date situational awareness picture, developing the correct course of action to 
respond to individual or multiple threats, and assigning the correct weapon to the right 
mission. 
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Figure 11. Find Fix Track Target Engage and Asses Dynamic Targeting Model (From 

Joint Publication 3-60, 2007). 


2. Dynamic Targeting Cell Responsibilities 

The DTC is an optional cell within the AOC and is only stood up under the 
control of the SODO when there are too many time sensitive targets for the regular AOC 
internal organization structure to process in a timely and effective manner. The DTC has 
a minimized chain of command and a streamlined F2T2EA standard operating procedure 
which supports quicker decision making and target prosecution times. During the TREX 
09-1, the DTC was responsible for completing the challenging task of prosecuting all 
time sensitive targets (TSTs) within the AOC area of responsibility. The DTC relies 
heavily on ISR and other information collected from the other internal AOC divisions but 
employs a ground track coordinator, an attack coordinator, a command and control duty 
officer, and a target duty officer. Eigure 12 shows the DTC structure in the dashed box. 
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Figure 12. Dynamic Targeting Center Structure (After Case, Koterba, Conrad, Okerman, 

& Vanderberry, 2006). 


During the TREX 09-1, the ground track coordinator was responsible for tracking 
and updating all TST locations. The attack coordinator and target duty officer were 
responsible for assessing the situation, selecting an appropriate weapon for the target 
selected, and planning the mission. The attack coordinator also accounted for verifying 
that the TST target was not on the no strike or no kill list (Shebilski, Freeman, Levchuk, 
& Gildea, 2008). The command and control duty officer drafts the air tasking orders for 
all TST missions. To assist with the communication and collaboration between the 
various DTC and AOC internal division personnel, the joint automated deep operations 
coordination system collaboration software application tool is used as the primary 
interface between divisions. 

3. Find Fix Track Target Engage and Assess Cognitive Processes 

The Air Force uses the F2T2EA, or “Kill Chain” as it is often referred to, to 
conduct dynamic targeting against TSTs. The kill chain provides AOC decision makers 
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the means by which to quickly engage targets of opportunity. According to the Joint 
Targeting publication (Joint Publication 3-60, 2007), TSTs are targets of very high 
importance to the Joint Force Commander’s mission and pose a significant threat to 
coalition forces. Our enemies today are aware of our tactics, procedures, and asset 
capabilities and use this knowledge to plot attacks against us. The enemy’s ability to 
quickly adapt, willingness to hide in caves for long periods of time, and readiness to use 
civilian populations as human shields, severely challenge the AOC commander’s 
cognitive processes and ability to effectively apply the F2T2EA targeting steps to counter 
TSTs. 


a. Step 1 “Find” 

The first step of dynamic targeting is “find.” Based on prior intelligence 
received, AOC commanders allocate ISR assets to search for TSTs. ISR assets such as 
airborne warning and control system aircraft, joint surveillance target attack radar system, 
unmanned aerial vehicles, and reconnaissance aircraft provide the AOC with the most up- 
to-date intelligence available. The ability to direct and task the often limited and the 
various types of ISR assets, when and where they are needed, requires skill and is a 
critical challenge for any commander conducting air operations. It can also be extremely 
difficult for those analysts responsible for analyzing and trying to decipher the usually 
large amounts of data. Because ISR assets are needed to support all types of missions, 
they are not always immediately available when needed. The cognitive processing ability 
of humans is limited which can make it difficult to process and synthesis the extremely 
large amount of data and information that is often received. To make matters worse, in 
order to be effective in a dynamic environment against TSTs, those processing the 
information must do so on a compressed time-line. The window of opportunity to 
prosecute TSTs is short and incomplete due to the style of warfare the enemy employs 
and the decision to act must often be made based on incomplete information. 
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b. Step 2 “Fix” 


Fix, the second step in dynamic targeting, refers to the ability to use 
available ISR assets to locate TSTs (Joint Publication 3-60, 2007). The process of fixing 
the target involves analyzing data by experienced personnel to provide an accurate 
picture of the battle space to support AOC commanders in assigning specific sensors at 
the correct time and intervals. This process can be a long and daunting task because 
information received from various sensors can often be ambiguous and incomplete. AOC 
commanders must have the necessary experience to recognize when they have enough 
information or what additional information they need to support their developing and 
achieving accurate situation awareness. This task can be a very stressful endeavor for 
those personnel responsible for asset management and/or making higher level decisions 
due to the high workload produced by needing to process large amounts of information 
under severely time compressed conditions with high consequences for failure. The 
decision maker must be able to de-conflict multiple air, land and sea assets, and 
coordinate and direct ISR assets to collect information on an enemy that is extremely 
gifted at concealing themselves. Juggling the multiple tasks can produce information 
overload for decision makers and analysts. Overload consequences can contribute to 
poor higher-level decisions and poor intelligence collection creating a loss of SA and 
resulting in greater casualties. 

c. Step 3 “Track” 

Track, the third step in dynamic targeting, starts after a fix is obtained. 
Assets are then assigned to continually observe and monitor the TST (Joint Publication 3- 
60, 2007). In a dynamic environment, a TST can change location within minutes. To 
keep assets tracking the target, decision makers and other personnel involved must 
maintain accurate SA to be able to immediately react to sudden changes. Maintaining SA 
requires the decision maker to keep track of multiple processes and events which can tax 
the operator’s working memory and creates a high cognitive workload. To make the 
right decisions, the decision maker needs to constantly share information and build new 
knowledge to stay abreast of the evolving situation. New knowledge is accomplished 
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through collaboration with other internal and external team members who are analyzing 
various data sets received from various sensors and then organizing the data into a 
meaningful structure. Dynamic interactions between personnel responsible for collection 
and those responsible for information processing must be continuous to ensure the correct 
sensors are assigned to the right type of targets at the precise time. AOC decision makers 
must have the ability to process a high volume of information under time-compressed 
conditions and then make time-critical responses to various situations in stressful 
environments (high workload, complex socio-technical environment and high 
consequences for errors). 

d. Steps 4 and 5 “Target and Engage” 

The fourth and fifth steps in the kill chain are target and engage. At this 
point, the AOC commander has the necessary information to engage and take appropriate 
actions to cause a desired effect (Joint Publication 3-60, 2007). Even with precise 
targeting information, other contributing factors, if not properly handled, can cause 
serious delays and jeopardize the mission. The final decision regarding whether or not to 
engage a target may also depend on the ability to de-conflict assets in the surrounding 
area to prevent “blue-on-blue” casualties, determination of rule of engagement 
authorizing type of attack, and assessment of risk of collateral damage to non-military 
personnel versus target value. At the conclusion of Operation Desert Storm, 25.6 percent 
of those killed were due to friendly fire (Krepinevich, 2003). Decision makers must be 
able to assess the impact of several additional factors on their decision before giving the 
order or permission to prosecute a TST. 

e. Step 6 “Assess” 

Assess is the sixth and final step in the kill chain. To be effective at this 
stage, the commander must base his/her decision on the following: how relevant and 
timely is the intelligence collected; are the right sensors being applied to the right target 
and at the right time; how effective are the weapons being employed against the TST; 
and is collateral damage being kept to a minimum? To answer these questions, the 


25 



commander must be able to direct ISR assets and allocate the correct sensors to conduct 
battle damage assessment. Overhead sensors from tactical and national assets provide 
imagery and other intelligence to help determine the overall state of TST targets after 
engagement. Based on the battle damage assessment, the AOC commander may decide 
that the target needs to be placed on the targeting list again or remove it completely from 
the TST prosecute list. In a dynamic environment, where the information can be 
incomplete, missing, or overwhelming, challenges to the human cognitive processing 
ability arise for the decision makers who at a moment’s notice must be able to effectively 
apply the kill chain against a TST (Joint Publication 3-60, 2007). 

D. TEAM COLLABORATION MODEL 

1. Previous Research 

Several previous theses have conducted research to evaluate earlier versions of 
the ONR developed model of team collaboration. Some of the first field research 
experiments regarding model evaluation focused on analyzing maritime interdiction 
operations (MIO) chat logs from three MIO exercises and air warfare audio transcripts 
from four different teams. A MIO is an operation that attempts to delay, disrupt or 
destroy an enemy’s supply resources before they can be used to harm people or cause 
other severe damage (Joint Publication 3-03, 2007). The air warfare scenarios involved 
team collaboration within a shipboard Combat Information Center to identify hostile and 
friendly air contacts (Hutchins, Kendall, Bordetsky, Bourakov, 2006). 

The first structural model of team collaboration validation thesis was done by 
Ensign Maura Garrity. The thesis involved analyzing the Fire Department of New York 
communication transcript that recorded several district and regional New York firefighter 
responses and actions in their response to the September 11, 2001, attacks on the World 
Trade Center. 

A second model of team collaboration thesis was conducted by Lieutenant 
Commander Catherine Donaldson and Lieutenant David Johnson. The thesis involved 
analyzing a transcript of recorded audio from the command and control center at 
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NORAD North East Air Defense Sector (NEADS) that recorded 
communications between NEADS, the Eederal Aviation Administration and other 
agencies during the events surrounding the discovery of the four hijacked aircraft on 11 
September 2001. 

A third structural model of team collaboration thesis was conducted by Eieutenant 
Euis Socias. The thesis analyzed a second communication channel between NEADS, 
Eederal Aviation Administration and other agencies during their response to the 
discovery of four hijacked aircraft on 11 September 2001. 
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III. LITERATURE REVIEW 


A. COGNITIVE PROCESSES 

The word cognition is derived from the Latin word cognoscere, which means 
“come to know” (Marian Webster Dictionary, 2009). An individual usually comes to 
know by developing and using his or her mental processes such as thinking, knowing and 
recollection to retain intellectual compilation and gain understanding. Cognition is a 
process that mainly occurs inside the human brain. The human brain is akin to a central 
processing unit of a computer. It is instrumental in taking in information from various 
sources, processing that information, and then storing the information in short-term and 
long-term memory. 

In addition to the basic functions of the brain, the human brain has four primary 
areas: working memory, attention and performance, visual spatial thinking, and learning 
recall and long term memory (Schifferstein & Hekkert, 2007). These primary areas are 
crucial to an individual’s ability to “come to know”. Gaining understanding by an 
individual or a group of individuals who are functioning together as a team is critical to 
the individual or team’s ability to effectively perform and solve complex, dynamic 
problems. Types of cognition include metacognition, macrocognition and 

microcognition. 

I. Metacognitive Processes 

Metacognition can be defined as an individual’s own understanding of his or her 
cognitive processes and that person’s ability to effectively use that unique understanding 
as knowledge to improve their cognitive processes (Schifferstein & Hekkert, 2007). In a 
nutshell, metacognition can be referred to as “thinking about thinking”. An example of 
metacognition would be that an individual notices that he or she is having a hard time 
learning event A compared to event B and therefore needs to double check event A to 
ensure comprehension. There are three classes of metacognition: (1) knowledge (i.e., 
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what an individual knows about their own cognitive processes); (2) regulation (i.e., an 
individuals own internal mechanisms used to control their cognitive learning); and (3) 
experiences (i.e., an individual’s own experiences with the current cognitive task) 
(Flavell, 1979). 

Individuals who have developed and can exercise sound metacognitive techniques 
are able to control and focus their learning processes. This can drastically improve their 
ability to work more effectively and perform increasingly more dynamic tasks. 
Metacognition is usually measured and evaluated in a laboratory environment by 
administering a pre-test before the event to test pre-existing knowledge. Then some of 
the individuals are shown metacognition techniques (Education Resources Information 
Center Digest, 1990). After the techniques are reviewed, all individuals start 
participating in the event or scenario. After the event is over, post-test are administered 
to all participants and the results between pre-test and post-test are analyzed to determine 
if metacognition techniques improved the individual’s knowledge level and/or over all 
event performance. 

2. Macrocognitive Processes 

Macrocognition is defined, for this research, as the internalized and externalized 
high-level mental processes employed by teams to create new knowledge during 
complex, one-of-a kind, collaborative problem solving (Letsky, Warner, Fiore, Rosen, & 
Salas, 2007). High-level mental processes refer to the cognitive processes involved in 
combining, visualizing, and aggregating information to resolve ambiguity in support of 
the discovery of new knowledge and relationships. Macrocognition studies concentrate 
on assessing complex problems of the naturalistic real world in contrast to 
microcognition studies which are typically focused and limited to laboratory type 
experiments. Some of the complex problems macrocognition studies include making key 
operational decisions that could affect the lives of individuals under poorly defined 
situations and under time pressures. Macrocognition focuses on both internal and 
external processes and how they affect the decision maker. Figure 13 shows 
macrocognitive functions and supporting processes that would be used by Dynamic 
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Targeting Cell (DTC) personnel in executing the Find Fix Track Target Engage and 
Asses (F2T2EA) process used while prosecuting time sensitive targets (TSTs). 



Eigure 13. Macrocognitive and supporting processes for individuals, teams, and 
information technologies (Erom Klein, Ross, Moon, Klein, Hoffman, & 

Hollnagel, 2003). 

3. Microcognitive Processes 

Microcognition is recognized as including the essential stepping stones for which 
thinking and processing are calculated and performed. Memory associated with 
microcognitive processes allow individual’s to cope with more dynamic situations and 
handle a greater complexity of information processing. Microcognition’s primary focus 
is to try to understand how the invisible processes that transpire inside an individual’s 
brain such as reasoning aptitude and mental processing capability occur and how they are 
directly affected either favorably or adversely (Klein, Ross, Moon, Klein, Hoffman, & 
Hollnagel, 2003). Some microcognition experts conduct experiments looking for 
communication design or team organizational model issues that can negatively impact an 
individual or team’s ability to complete assigned tasks. Poor designs can make it harder 
on individuals to retrieve important task relevant knowledge (i.e., knowledge bottleneck); 
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remember or store into memory specific key task related details (i.e., memory 
bottleneck); and/or make it more difficult to stay focused on the task at hand or switch 
back and forth from different tasks (i.e., attention bottleneck) (Schifferstein and Hekkert, 
2007). Figure 14 shows the situational design interface representing cognitive processes 
involved. In order to assess microcognition, eye tracking sensors and other measurement 
devices must be properly designed, implemented, and assessed. The study of 
microcognition is typically done in a laboratory environment with well defined attributes 
and goals. 



LonK-tt-rm .Mcmor\ 
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Figure 14. Situational Design and Cognitive Model Process (From U.S. Department of 

Transportation, 2009). 


B. TEAM PERFORMANCE 

1. Performance Factors 

Performance is defined as an individual’s ability to execute one or several tasks 
with a specific end result (Marian-Webster Dictionary, 2009). There are several internal 
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and external faetors that affeet an individual or a group’s ability to earry out specifie 
tasks in support of unified missions. Internal faetors that affect performance include 
individual feelings, emotions, skill level, and inherent core values. External factors 
include the organizational structure hierarchy (i.e., distributed team, centralized team, 
etc...), and situational environment factors (i.e., time, space, force and other dynamic 
elements). Another key factor that can affect performance is the common understood 
situational awareness between individual teams and team members. In a highly dynamic 
environment supporting multiple theater operations, it is very easy for an individual to 
become inundated with too much information leading to information overload resulting 
in poor decision making. Figure 15 is a performance topology map that shows the 
relationship between both internal and external factors that affect performance. 



Figure 15. Performance Topology Map (From Clark, 2004). 
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2. 


Team Performance Metrics and Improvements 


Performance metrics can be established by identifying all the critical processes for 
accomplishing the individual or team mission objective(s). Next standard operating 
procedures are developed that are used to accomplish ah the identified critical processes. 
Finally, a value is assigned to each accomplished process based off some variable related 
to the task (i.e., time of completion, accuracy rating, communication percentage). These 
values can then be used as a baseline score that can be compared to other changes made 
to the system in order to determine if changes in the organizational structure, processing 
procedures, or human interaction resulted in positive or negative results. 

Overall performance can be drastically improved by ensuring that ah individuals 
understand the operational mission objectives and performance goals. Each member 
should be briefed on common factors to ensure individual mental models are consistent 
with one another to ensure everyone interprets the goals with the same mind set. 
Additionally, continued metric studies of the organizational hierarchy and standard 
operating procedures should be done on a regular basis to make certain the best structural 
model and procedures are being used to accomplished goals (Nikols, 2003). 

C. DECISION MAKING 

I. Decision-making Methods 

Research in decision making is a long and ongoing process that continues to look 
at how people make decisions. There are various methods and combinations of decision¬ 
making techniques that affect how decisions made. The traditional method of decision 
making continues to be one of the preferred methods consistently used in both civilian 
and military environments. According to Orasanu and Connolly, traditional decision 
makers systematically go through ah possible alternatives before choosing the course of 
action that offers the best outcome. This method for making decisions allows the 
decision maker to choose their final choice based on known goals, purposes, and values 
(Klein, Orasanu, Calderwood, & Zsambok, 1993). A more recent decision making 
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method has been described by Dr. Gary Klein called recognition-primed decision 
making. Klein believes that people make decisions much faster and without comparing 
the choices as shown in studies of both naturalistic decision making and recognition- 
primed decision making. 

2. Classical Approach 

Classical decision making is a cognitive progress by which personnel in decision 
roles follow a step-by-step procedural and/or repeatable way of coming up with the best 
choice for a given situation. Decision makers arrive at what they feel is the best solution 
by logically going through a list of choices and weighing the pros and cons of each option 
and decision before settling on a final choice. The process involves: identifying a set of 
rules and boundaries of the problem; establishing a procedure or system to evaluate all 
options; weighting each option outcome against all other option outcomes; and finally, 
performing a grading of the options. The decision maker will pick the option that 
provides them with the highest or lowest score depending on how grading rules were 
setup (Klein, 1999). 

According to Beach and Lipshitz, many decision makers resist this classical 
decision making method because it is cumbersome and takes lots of the decision maker’s 
time (Klein, et ah, 1993). In an experiment conducted at Massachusetts Institute of 
Technology Sloan School of Management, students in their final year of college were 
studied to see what decision strategies they would use to select their future job. 
Surprisingly, students choose to go with their gut feelings rather than sequentially going 
through the steps to come up with the best choice (Klein, 1999). A similar observation of 
firefighter commanders, by Klein, operating in stressful environments concluded that 
firefighters with over twenty years of experience behaved in the same manner as students 
in the MIT experiment (Klein, 1999). 

Although decision makers may use the traditional systematic method in which 
traditional decision making is performed, classical decision making remains a method 
that is widely used and is extremely useful in various situations. In a strategic 
environment, where traditionally time and speed are not critical factors, the classical 
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approach allows the decision makers to take a step back, look at all the different 
operational and tactical players involved, consider the various military and political 
options and implications, weigh the choices against one another, and finally pick the best 
choice that would minimize casualties and guarantee mission success. Classical decision 
making also works well at commands that have primary missions of dictating and writing 
policies. 

In a dynamic environment, such as an AOC conducting missions over Iraq and 
Afghanistan, the classical approach will not work. AOC commanders do not have the 
time to fully run through all the different possible options before deciding on a course of 
action. The commander’s window of opportunity to engage high-value targets could last 
for only a few minutes or even only seconds which means the commander must have the 
necessary training and experience to rapidly decide what are the best techniques and 
assets to deploy for a given target. Klein referred to decisions made in a time limited 
environment naturalistic decision making. 

3. Naturalistic Approach 

Naturalistic decision-making is the “study of how people use their experience to 
make decisions in field settings” (Klein, 1999). Unlike classical decision making, where 
decision makers follow an orderly process before finalizing their decision, during 
naturalistic decision making, decision makers access past experience that applies to the 
current situation. According to Klein, time pressure, high stakes, experienced decision 
makers, inadequate information, ill-defined goals, poorly defined procedures, cue 
learning, context, dynamic conditions, and team coordination are all features that set 
naturalistic decision making apart from classical decision making (Klein, 1999). Human 
decision making research conducted by Orasanu and Connolly leads us to believe that the 
above decision making features (i.e., time pressure) have been left out of past decision 
making research resulting in an incomplete view into the human decision making process 
(Klein, et ah, 1993). 
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An AOC commander conducting time-sensitive targeting faces some if not all of 
the above naturalistic decision making features. The commander, based on his or her 
level of experience, must quickly decide on asset allocation and weapons employment for 
a given target. Klein estimated that firefighter commanders make 90 percent of their 
decisions in less than one minute (Klein, 1999). The experience level of the commander 
plays a major role in their decision making capabilities. Additionally, past mission goals 
and priorities, rules of engagement limits, reliable intelligence, and predictable casualty 
rates are other features of importance to the commander. These other features are stored 
within an experienced commander’s memory and are quickly recognized and used in the 
commander’s decision making process. Klein refers to this as recognition-primed 
decision making. 

4. Recognition-Primed Decision 

Klein, while working with firefighter commanders, noticed that experienced 
firefighters in a given situation will immediately recognize the most critical factors and 
then decide and implement the best course of action. The past experience of the 
commanders immediately takes over and without having to go through a list or 
comparing possible options, the commander is able to decide and act. To an experienced 
decision maker, the classical way of coming up with a decision wastes valuable time 
which may result in more lives lost and could be the difference between winning or 
losing the fight. Klein believes that recognition prime decision making is a more 
strategic and clever way of using one’s own experience to quickly come to a decision 
(Klein, 1999). 

Recognition prime decision making is the fusion of a decision maker’s use of 
experience to size up a given situation in order to come up with a course of action and the 
manner in which he or she mentally simulates implementation of the course of action. 
The observations and interviewing of experienced firefighters and military decision 
makers, under time stressed environments, provided Klein with conclusive data on how 
people use experience to make decision. Klein believes that a decision maker’s 
experience determines his or her course of action and such a decision is normally made 
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without the use of classical decision making techniques (Klein, 1999). Figure 16 is 
Klein’s model on recognition prime decision making. The model shows how an 
experienced decision maker mentally steps through developing a single course of action. 


Variation 3 

Evaluate Course ot Action 



Figure 16. Recognition Prime Decision Model (From Klein, 1993). 
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IV. MEASUREMENT MODEL EOR MACROCOGNITION 

RESEARCH 


A. MODEL MACROCOGNITION FOCUS 


The concept of macrocognition is to develop a better understanding of the 
cognitive processes involved when a team collaborates to solve a unique, complex, time- 
compressed problem (Letsky et ah, 2007). Figure 17 is the team measurement model for 
macrocognition research that shows the relationship between different phases of 
knowledge building and developing understanding during team collaboration. This 
measurement model was developed to try and capture and measure the macrocognitive 
processes. The term macrocognition was coined to capture and distinguish higher-level 
cognitive processes used by individuals and teams from lower-level, or microcognitive 
processes (e.g. attention, perception, and memory) (Letsky et ah, 2007). 



Legend: 



External 

Data 



Output(s) 


Figure 17. Team Measurement Model for Macrocognition Research (From Fiore, et ah, 

in press) 


39 












































To better understand the relationship between the different phases of the 
measurement model, shown in Figure 17, one must understand that there is not initial 
model starting point. Different team dynamics (i.e., personnel, asynchronous team) and 
assigned tasks will dictate what phase of the model is used. The focus of macrocognition 
is on building new knowledge in real-word settings when a team collaborates. Figure 18 
illustrates how unstructured raw pieces of datum are transformed into integrated 
actionable knowledge. 



Understanding 

Figure 18. Knowledge Building Process in Macrocognition (From Fiore, et ah, in press) 

Macrocognition is demonstrated during the conversion of internalized team 
knowledge into externalized team knowledge through the individual knowledge and team 
knowledge building processes (Fiore, et ah, in press,). Data is considered to be raw and 
unprocessed bits of particulars. For data to be transformed into information it must be 
organized and referenced in some context thus being considered processed information. 
Information is considered to be knowledge when it is organized in such a way that allows 
it to be understood and used to solve a problem or direct actions. Table 2 provides a 
formal definition, example and explanation of data, information and knowledge. 
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Table 2. 


Data, Information, and Knowledge Definitions (From Fiore, et al., in 

press). 


Concept 

Explanation 

Data 

Definition 

Data are disparate statements or facts presented or represented 
separately and without context. 

Example 

“The CH-53 Marine Corps helicopter can hoist 250 feet with a 600 
pound lift capacity.” 

Explanation 

Here the content is devoid of context and not organized in any way; as 
such, it is considered only data. 

Information 

Definition 

Information is organized or structured data (i.e., organized or 
structured statements or facts) that have been related to the problem 
solving context. 

Example 

“The CH-53 Marine Corps helicopter can carry supplies to the Red 
Cross workers who have been taken hostage,” or, “Our three air 
vehicles are the CH-53, the F-16, and the E2-C.” 

Explanation 

The first example represents a transformation in that it involved 
connecting the piece of data to the problem. The second example 
represents a transformation in that it involved organizing the data via 
categorization such that it can serve the problem solving; resources 
were organized into categories of resources that serve the problem 
solving. 

Knowledge 

Definition 

Knowledge is the integration of content from two or more categories of 
information into something which did not explicitly exist before and 
which has been made actionable by being related to the problem 
solving context. 

Example 

“The CH-53 Marine Corps helicopter cannot be used to carry supplies 
because it is foggy over the southwest corridor of Nandor.” 

Explanation 

This represents a transformation because vehicle information (a 
category), was integrated with weather information (another category) 
in such a way that it serves the problem solving; that is, it was made 
actionable by explaining when it could get (or not get) supplies to the 
hostages. 


B. MACROCOGNITIVE PROCESS DEFINITIONS 


The focus of the macrocognitive process model and definitions is to measure all 
macrocognitive processes that occur during a collaborative team problem solving task. 
The measurement model attempts to understand the related processes between 
internalized knowledge, individual knowledge building processes, team knowledge 
building processes and externalized team knowledge macrocognitive phases 
demonstrated during team collaboration. Data on internalized knowledge is measureable 
by using eye tracking equipment and conducting calculations on eye gaze. Individual 
knowledge building data is measurable by collecting observation information. Data on 
team knowledge building processes is collected through communications and non-verbal 
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communication gestures (i.e., facial expressions). Externalized team knowledge is 
collected through communications and actual objects that are created by the team (i.e., 
maps, charts, and graphs) (Fiore, et ah, in press). The increased perception and 
understanding could one day allow researchers to be able to accurately predict the 
individual and/or group of teams, generated problem solving outcomes. 

C. STAGES OF THE MEASUREMENT MODEL 

The team measurement model was developed from various iterations of other 
previous macrocognitive models. The model focuses on measuring the macrocognitive 
processes engaged in by the team as they build knowledge at the individual and team 
levels (Fiore, et ah, in press). Team collaboration within the measurement model consists 
of five macrocognitive stages that show how members collaborate with each other and 
with the team as a whole. In certain stages of the model, information can flow back and 
forth between each other or skip stages all together. There is no official macrocognitive 
building block or individual or team information processing starting point for the team 
measurement model stages as theoretically a team member can start in any model stage. 
For the purpose of describing the model stages and their interactions, in this thesis we 
will start by describing the flow of individual and/or team information processes at the 
internalized team knowledge stage flowing into the individual knowledge building stage 
unto team knowledge building, externalized team knowledge, and team problem solving 
outcomes. Further explanation of each cognitive stage and their associated cognitive 
processing codes are defined in Table 3. 


42 



Table 3. 


Macrocognitive Stages and Associated Processes included in the 
Measurement Model (From Fiore, et al., in press). 


1 

stage I: Internalized Team Knowledge Process: Refers to the collective knowledge held in the individual 
minds of team members. Internalized Team Knowledge is measured by eliciting it from individual team 
members using methods such as card sorting, concept mapping, paired comparison ratings, scenario probes. 

a. 

Team Knowledge Similarity: Team knowledge similarity can involve the degree to which differing roles 
understand one another (e.g., how well a land/sea vehicle specialist understands a humanitarian specialist), or 
how well the team members’ understand the critical goals and locations of important resources (shared situation 
awareness). 

b. 

Team Knowledge Resources: Team members’ collective understanding of responsibilities and resources 
associated with the task. 

2 

Stage II: Individual Knowledge Building Process: is a process which includes actions taken bv individuals 
in order to build their own knowledge. These processes can take place inside the head (e.g., reading, mentally 
rotating objects) or may involve overt actions (e.g., accessing a screenshot). 

a. 

Individual Information Gathering: Individual information gathering involves actions individuals engage in to 
add to their existing knowledge such as reading, asking questions, accessing displays, etc. 

b. 

Individual Information Synthesis: Individual information synthesis involves comparing relationships among 
information, context, and artifacts to develop actionable knowledge. 

c. 

Knowledge Object Development: Knowledge object development involves creation of cognitive artifacts that 
represent actionable knowledge for the task. 

3 

Stage III: Team Knowledge Building Process: is a nrocess which includes actions taken bv teammates to 
disseminate information and to transform that information into actionable knowledge for team members. 

a. 

Team Information Exchange: Team information exchange involves passing relevant information to the 
appropriate teammates at the appropriate times. 

b. 

Team Knowledge Sharing: Team knowledge sharing involves explanations and interpretations shared between 
team members or with the team as a whole. 

c. 

Team solution Option Generation: Team solution option generation describes offering potential solutions to a 
problem. 

d. 

Team Evaluation and Negotiation of Alternatives: Team evaluation and negotiation of alternatives describes 
clarifying and discussing the pros and cons of potential solution options. 

4 

Stage IV: Externalized Team Knowledge Process: Refers to facts, relationshins, and concents that have been 
explicitly agreed upon, or not openly challenged or disagreed upon, by factions of the team. 

a. 

Externalized Cue-strategy Associations: Externalized cue-strategy associations describe the team’s collective 
agreement as to their task strategies and the situational cues that modify those strategies (and how). 

b. 

Pattern Recognition and Trend Analysis: Pattern recognition and trend analysis is the accuracy of the patterns 
or trends explicitly noted by members of a team that is either agreed upon or unchallenged by other team 
members. 

c. 

Uncertainty Resolution: Uncertainty resolution is the degree to which a team has collectively agreed upon the 
status of problem variables (e.g., hostile/friendly). 

5 

Stage V: Team Problem Solving Outcomes: Are assessments of aualitv relating to a team’s problem solutions 
or plan. 

a. 

Quality of Plan: Quality of plan (problem solving solution) involves the degree to which the solution adopted 
by a problem solving team achieves a resolution to the problem (e.g., limit fatalities, limit destruction). 

b. 

Efficiency of Planning Process: Efficiency of planning process describes the amount of time it takes a 
problem solving team to arrive at a successful resolution to a problem. 

c. 

Efficiency of Plan Execution: Efficiency of plan execution describes the quality of the plan (e.g., number of 
lives saved) divided by the amount of resources used to accomplish this and the amount of time the plan takes to 
unfold. 
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1 . 


Internalized Team Knowledge 


The first stage of the measurement model for macrocognition research focuses on 
Internalized Team Knowledge (ITK) which refers to the overall combined knowledge of 
each team member (Fiore, et ah, in press). ITK is collected on individuals by conducting 
various assessments such as administrating tests and surveys to gather past individual 
experience. The knowledge collected on individuals provides data on the level of team 
knowledge at different points in time which provides insight into the knowledge building 
process. 

2. Individual Knowledge Building 

The second stage of the measurement model focuses on Individual Knowledge 
Building (IKB). The cognitive process of team members can flow to IKB or to team 
knowledge building (TKB). IKB actions are actions such as reading or asking questions 
that can be taken by an individual to increase his or her knowledge (Fiore, et ah, in 
press). An individual or the team as a whole asking for more information is a sign that an 
individual or team members are trying to improve individual situational awareness. To 
achieve situational awareness, members must take immediate action to correct 
information deficiency and gain knowledge. To increase their knowledge, team members 
can either ask other team members for help or obtain outside specific job related 
knowledge building from schools or training. As the measurement model shows, the 
feedback loop allows members to seek outside knowledge building. 

3. Team Knowledge Building 

The third stage. Team Knowledge Building (TKB), is a highly dynamic and 
iterative process. TKB facilitates information exchange among teammates with the intent 
to generate a plan or coordinate some type of action (Fiore, et ah, in press). Actionable 
information will be processed and disseminated as a solution to team related problems 
and non-actionable information will remain in the minds of team members as internalized 
knowledge. A similar process takes place within an AOC targeting cell. Information 

received on a high-value target may not be used to perform targeting mission. Some 
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information will be incomplete and require validation through collection from overhead 
or ground assets. Until further information is collected, unused data will be stored in 
databases until more information becomes available. 

4. Externalized Team Knowledge 

The fourth stage, Externalized Team Knowledge (ETK) is defined as knowledge 
agreed upon by team members to be factual (Eiore, et ah, in press). Knowledge held by 
team members is different from information, because unlike information, agreed upon 
knowledge is put through processes to ensure accuracy and completeness (Eiore, et ah, in 
press). Such processes could range from verifying sources to comparing output from 
multiple intelligence, surveillance and reconnaissance assets. The intelligence 
community uses a similar process to ensure information received from various 
intelligence sources is accurate. Data collected goes through a validation process and is 
analyzed before the information can be considered as intelligence. 

5. Team Problem Solving Outcomes 

During the Team Problem Solving Outcomes stage potential solutions are 
assessed to determine if they meet certain criteria of effectiveness (Eiore, et ah, in press). 
In a naturalistic environment, AOC target cell members conduct battle damage 
assessment to measure the effectiveness of weapons used. According to the 
commander’s handbook for Joint Battle Damage Assessment, battle damage assessment 
is the “timely and accurate estimate of damage resulting from the application of military 
force, either lethal or nonlethal, against a predetermined objective.” By using various 
combinations of intelligence, surveillance and reconnaissance assets, commanders can 
measure the performance of their forces and weapons effectiveness against different 
targets. 
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V. METHOD 


A. EXERCISE DATA SELECTION 

The TREX 09-1 exercise was conducted 20-25 October 2008, with exercise 
scenarios conducted in four-hour sessions. On 22 and 23 October, two sessions were 
executed on each day (one AM and one PM session). On October 24, only one session 
was conducted and on 25 October, exercise personnel were debriefed. 

Both authors, in conjunction with the thesis advisor determined that the best data 
for conducting analysis and coding to empirically evaluate the model of team 
collaboration would be the data generated during the middle four sessions. Our reasoning 
was based on our expectation that there would be a learning curve on day one as 
personnel were becoming familiar with the exercise operating environment, procedures, 
and other exercise personnel operators. 

B. DATA FORMATTING 

The TREX 09-1 chat logs were presented to us spread out in several tables located 
in one access database file that was generated from the Joint Automated Deep Operations 
Coordination System (JADOCS). JADOCS is a mission management software program 
that enables integration and sharing of information and common operating picture 
(Jane’s, 2009). Exercise chat logs included 15 different internal chat rooms. Both 
authors extracted the Microsoft Internet Relay Chat log data from a JADOCS database 
access file. Data was pulled from the fifteen different chat logs for the (4) four-hour 
exercise sessions selected for coding. The data was then imported into a Microsoft excel 
document with four tabs, each labeled and containing the data from the four sessions 
selected for analysis. Each data set was then organized by specific chat room and then 
ordered by time of transmission (earliest to latest). Each chat log entry also contains 
originator and destination chat room information. 
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C. PRACTICE CODING 

Practice coding was completed by both coders utilizing the revised set of 
macrocognitive process included in the model of team collaboration on 1000 lines of 
maritime interdiction operation exercise data. Both coders began by coding 100 lines of 
chat log data separately and then reviewed the coding with the thesis advisor. This 
initial practice coding and review of additional practice coding on a separate transcript 
was considered sufficient training to calibrate the coders on the coding process. As a 
team, we reviewed additional lines of coding along with the definitions and focused the 
discussion on where coders disagreed on their coding of individual chat log entries. After 
this review of the initial coded data, the rest of the maritime interdiction operation data 
was coded separately by the two authors and then reviewed to help ensure the two coders 
were well calibrated. 

D. FINAL CODING OF TRANSCRIPTS 

All four TREX 09-1 exercise sessions were coded separately by the coders. The 
first 100 lines from 22 October AM data set were coded first. Once completed, the 
coding was discussed and reviewed with the thesis advisor to ensure macrocognitive 
process definitions were being applied in a consistent manner. Following this final 
review and calibration, both coders coded the rest of the 22 October AM session and 
reviewed all coded data to ensure a consistent interpretation of the macrocognitive 
process definitions. During the review, new codes and definitions that were difficult to 
interpret/apply were discussed and justified to re-confirm the coding process. The 
subsequent remaining three exercise sessions were coded separately and then reviewed at 
the completion of each session prior to moving onto a new session. Again, this was done 
to discuss any new codes assigned and to discuss the more difficult speech turns. This 
rigorous process was employed to ensure the TREX 09-1 exercise team communications 
were interpreted correctly and to help ensure a consistent application of the 
macrocognitive process definitions. 
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E. MEASURE OF INTER-RATER RELIABILITY 

Both authors coded 2,493 lines of chat log entries contained within the four 
exercise sessions. Once the coding was completed, both coders reviewed the data again 
to try to resolve differences in some codes assigned to data. After thorough and diligent 
deliberation, there were some originally assigned codes that were changed to match the 
other coder’s code but not all codes match as there still remained some differences of 
opinion between code definition interpretations. To determine the overall percentage of 
agreement between the two coders, the qualitative categorical statistic Kappa Cohen was 
used to calculate the percentage of agreement between the two coders using the code 
values assigned by each coder. An additional code was created called the Extra Code 
Filler (ECF). The ECF code was necessary to ensure that each coder assigned the exact 
same number of total codes in order to accurately calculate inter-rater reliability for the 
two coders, using Kappa Cohen. Kappa Cohen is the preferred statistic over the Chi- 
square statistic as Kappa Cohen tests for agreement where as Chi-square tests for 
association (Thomas, & Hersen, 2003). Kappa Cohen is a better statistic for measuring 
categorical items as it accounts for and factors into the calculation that each coder may 
also agree by chance and not strictly because they chose the same selection option or 
code. Figure 19 shows the Kappa Cohen statistic. In the Kappa Cohen equation, Pr(a) 
value is the observed agreement among coders and Pr(e) is the hypothetical probability of 
chance agreement. 

, Pr(d)-Pr(e) 

k= - 

1-Pr(e) 

Figure 19. Kappa Cohen Statistical Equation (From Wikipedia, 2009). 

Kappa Cohen has a range of 0 to 1. The larger the value calculated indicates a 
greater agreement between the two coders. A Kappa Cohen value of .0 to .20 indicates a 
slight agreement, .21 to .40 indicates a fair agreement, .41 to .60 indicates a moderate 
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agreement, .61 to .80 a substantial agreement, and .81 to 1 indicates an almost perfect 
agreement. A Kappa Cohen value of 0 indicates no agreement between coders 
(Wikipedia, 2009). 

To calculate Kappa Cohen, both coders organized the exercise data so there was 
only one code assigned to each line of chat log text. If one coder did not code a specific 
line of data, that is, if there was a disagreement between the coders on whether a code 
should have been assigned, the ECF code was used to ensure each coder assigned the 
same total number of codes. The macrocognitive process definitions were assigned a 
numerical value for each code (i.e.. Administrative = 1, Miscellaneous = 2, Team 
Information Exchange = 3, etc.) and then a 15x15 contingency table was filled out using 
the coder assigned codes. A 15x15 table was needed and used during our coding process 
as we assigned 15 different codes to our data. Figure 20 shows a sample 3x3 contingency 
table to illustrate the calculation process: 


Rater #1 


Rater #2 

"y" 

II 

"c" 

"y" 

9 

3 

1 

II *» 

4 

8 

2 

"c" 

2 

1 

6 


Figure 20. Sample 3x3 Contingency Table (From University Nebraska 2009). 

In Figure 20, the diagonal cells of the matrix indicate agreement between the 
coders whereas the other cells and associated values indicate the difference between what 
each of the coders chose. In the matrix, both raters agreed 9 times on “y”, 8 times on 
“r”, and 6 times on “c”. By totaling column 1 and row 1, you can deduce that rater 1 
selected “y” 15 times and rater 2 chose “y” 13 times. By totaling all the column values 
or rows you can find the total number of codes assigned per rater. In the matrix above, 
36 codes were assigned. 
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To determine Pr(a) in the Kappa Cohen equation for the above matrix, one takes 
the diagonal values 9, 8, and 6 (sum 23) and divides by 36 which is the total number of 
codes assigned, where the result is Pr(a) = .6389. To find Pr(e), one must determine the 
percentage of times rater 1 and rater 2 chose “y”, “r” and “c” individually and multiply 
the “y”, “r”, and “c” percentages against one another and sum the results. For the above 
matrix, rater 1 chose “y” 15 times out of 36, resulting in .4167 percent, “r” 12 times out 
36, resulting in .3333 percent, and “c” 9 times out of 36, resulting in .25 percent. Rater 2 
chose “y” 13 times out 36, resulting in .3611 percent, “r” 14 times out 36, resulting in 
.3889 percent, and “c” 9 times out of 36, resulting in .25 percent. Rater 1 “y” times rater 
2 “y” = .1505. Rater 1 “r” times rater 2 “r” = .1296. Rater 1 “c” times rater 2 “c” = 
.0625. By adding up the percentages .1505, .1296, and .0626, Pr(e) equal .3426. 
Therefore, k is calculated as .4507, which indicates rater 1 and rater 2 have a moderate 
inter-rater reliability of agreement. 
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VI. RESULTS 


A. CODING RESULTS 

1. Code Definitions and Interpolation 

The TREX 09-1 exercise was analyzed and coded as four individual event 
sessions. The analyzed classified appendix 1 data is retained in the classified section of 
the Naval Postgraduate School Dudley Knox library. The first session’s data contained 
528 entries of text, session two through contained 625, 668, and 672 entries of text, 
respectively. Total entries of text analyzed and assigned codes were 2,493. Session text 
consisted of anywhere from a 1 word statement to a 40 word paragraph. Most entries, 
based on macrocognitive complexity and number of utterances, were assigned multiple 
codes. For the 2,493 lines of text analyzed, 3,158 codes were assigned by each coder. 
These codes represent the cognitive processes used during the TREX 09-1 exercise. Each 
coder analyzed the text one line at a time and assigned a code for each line of text. This 
process was necessary, along with the use of the Extra Code Filler codes to ensure equal 
distribution of codes between coders when there was a disagreement on whether a code 
should or should not be assigned, to facilitate the calculation of inter-rater reliability. 
TREX 09-1 coded speech turn examples are listed in Table 4 along with their associated 
measurement model code definition. 

Table 4. Measurement Model Macrocognitive Process Code Definitions and TREX 

09-1 Coded Examples. 


1 

Stage I: Internalized Team Knowledge Process: Refers to the collective knowledge held 

in the individual minds of team members. Internalized Team Knowledge is measured by eliciting it 

from individual team members using methods such as card sorting, concept mapping, paired 

comparison ratings, scenario probes. 


Team Knowledge Similarity: Team knowledge similarity can involve the degree to which 

differing roles understand one another (e.g., how well a land/sea vehicle specialist understands a 

humanitarian specialist), or how well the team members’ understand the critical goals and locations 

of important resources (shared situation awareness). 
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- No Coded examples for AOC data 


Team Knowledge Resources: Team members’ collective understanding of 

resources/responsibilities associated with the task. 

-1 remember sketchy authentication codes 

- Fighter aircraft #2 is out of position, looks like other strike assets are quicker 

- He wouldn’t request to return to base (RTB), he tells you he is RTB 

2 

Staae II: Individual Knowledae Building Process: is a process which includes actions 

taken by individuals in order to build their own knowledge. These processes can take place inside 

the head (e.g., reading, mentally rotating objects) or may involve overt actions (e.g., accessing a 

screenshot). 


Individual Information Gathering: Individual information gathering involves actions 

individuals engage in to add to their existing knowledge such as reading, asking questions, accessing 

displays, etc. 

- What is the correct way to pass tasking to a predator to attack? 

- Joint coordinating elements do you know the local threat/ risk in the area 

and do you have imagery of the locations? 

- Any battle damage assessment reports/imagery post-strike for aircraft? 


Individual Information Synthesis: Individual information synthesis involves comparing 

relationships among information, context, and artifacts to develop actionable knowledge. 

- Reliable sources report a known country bomb component supplier is 

awaiting a large shipment of explosives 

- It is suspected that a certain country uses this location as a storage facility 

for spent fuel. 


Knowledge Object Development: Knowledge object development involves creation of 

cognitive artifacts that represent actionable knowledge for the task. 

- No Coded examples for AOC data 

3 

Stage III: Team Knowledge Building Process: is a process which includes actions taken 

by teammates to disseminate information and to transform that information into actionable 

knowledge for team members. 


Team Information Exchange: Team information exchange involves passing relevant 

information to the appropriate teammates at the appropriate times. 

- Target priority coordinated, entered and pushed to joint time sensitive 

targeting manager 

- The actual snatch and grab would be possibility for special operation 

force (SOF) but we would need intelligence assistance 

For your information, this area is now under SOF control. Reconnaissance 
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aircraft to provide over watch, SOF is now in contact with aircraft 


Team Knowledge Sharing: Team knowledge sharing involves explanations and 

interpretations shared between team members or with the team as a whole. 

- Self defense applies for hostile acts from one country airspace to another 

- Enemy forces that employ ordnance, electronic attack or achieve a radar 

lock against friendly forces have committed a hostile act. 


Team Solution Option Generation: Team solution option generation describes offering 

potential solutions to a problem. 

- Awaiting radiological impact assessment on watershed if strike building. 

Second option in work is destroy local roads to prevent access in/out. 

- If we crater the runway and taxiways, we may be able to effectively stop the target. 

- To shorten timeline for tactical tomahawk we can launch to loiter. 

will attempt to mitigate with weaponeering 


Team Evaluation and Negotiation of Alternatives: Team evaluation and negotiation of 

alternatives describes clarifying and discussing the pros and cons of potential solution options. 

- Just throwing this out there, but if you target the roadways, is there a 

chance you could spook them and they might fire off their missiles and run? 

4 

Stage IV: Externalized Team Knowledge Process: Refers to facts, relationships, and 

concepts that have been explicitly agreed upon, or not openly challenged or disagreed upon, by 

factions of the team. 


Externalized Cue-strategy Associations: Externalized cue-strategy associations describe 

the team’s collective agreement as to their task strategies and the situational cues that modify those 

strategies (and how). 

- The dynamic effect cell chief stated that if there is an erect launcher in a 

joint special operations area the "rule of engagement" is to kill it as soon 

as possible and if there is time to de-conflict with the teams 

He mentioned tomahawk land attack missile (TEAM) wouldn't be de-conflicted 

either, but I dispute that logic. First, we wouldn't use a TEAM shot to kill a launcher 

I don't think. Unless it was a last resort. 

- Can get special operation force Team to location as additional resource if we elect 

to monitor the site for any potential leadership meetings that may occur later 


Pattern Recognition and Trend Analysis: Pattern recognition and trend analysis is the 

accuracy of the patterns or trends explicitly noted by members of a team that is either agreed upon or 

unchallenged by other team members. 

- Looks like this target may be similar to our first target with regards to unknown 

presence of Radiological containers in facility. We would look at interdiction for 
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containment to prevent travel to/fm that site, your thoughts on best plan/option 


Uncertainty Resolution: Uncertainty resolution is the degree to which a team has 

collectively agreed upon the status of problem variables (e.g., hostile/friendly). 

- Tomahawk land attack missile most definitely have to be de-conflicted even for over 

flight of the joint special operations area unless direct otherwise by the Joint Force 

Commander 

5 

Staae V: Team Problem Solving Outcomes: Are assessments of aualitv relating to a 

team’s problem solutions or plan. 

- No Coded examples for AOC data 


Quality of Plan: Quality of plan (problem solving solution) involves the degree to which 

the solution adopted by a problem solving team achieves a resolution to the problem (e.g., limit 

fatalities, limit destruction). 

- No Coded examples for AOC data 


Efficiency of Planning Process: Efficiency of planning process describes the amount of 

time it takes a problem solving team to arrive at a successful resolution to a problem. 

- No Coded examples for AOC data 


Efficiency of Plan Execution: Efficiency of plan execution describes the quality of the 

plan (e.g., number of lives saved) divided by the amount of resources used to accomplish this and the 

amount of time the plan takes to unfold. 

- No Coded examples for AOC data 


During our coding of the data, both coders came across certain speech turns that 
did not adequately represent one of the definitions in the measurement model. To 
properly represent the data via assigning codes, it was necessary for us to use non¬ 
measurement model codes to represent certain speech turns. Table 5 lists the non¬ 
measurement model codes and associated definitions that were necessary to cover gaps 
between the measurement model codes and the session data. 
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Table 5. Non-Measurement Model Codes/Definitions and TREX 09-1 Coded 

Examples. 



Administration: Codes necessary for exercise support but not relevant or pertinent to the exercise 
scenarios. 

- Test 

- In the future, please post reports in “InteTReport” 

- Cancel that 

- Command and control information posted to folder 

- Chat check 


Miscellaneous: Codes that did not include a macrocognitive process but were part of normal 
closed-loop communications. 

- Copy, please note request for information 

- Roger on target location 

- Roger, thank you 

- SIDO, standby, checking 

- Copy and standby for additional informational 


Course of Action: Action given that when implemented will significantly affect the scenario 
outcome. 

- Contact fighter aircraft #12 on circuit #2 for clearance to drop weapons. 

- Move aircraft to provide over watch for Special Operation Force teams 

- You can move fighter aircraft #2 and #10 to training camp located in the vicinity of 
of city #3 and city #5. Upon completion of mission, return to current location 

- Plan is to strike unless directed otherwise 

- Move aircraft to investigate lED implantation report 

4 

Reauest to Take Action: Lower-level action request between peers that most likely would not 
affect the scenario outcome. 

- Please instruct aircraft #1 to observe possible SCUD hiding site 

- Please pass report to all 

- Need you to check with air combatant commander and special operations 
commander for teams in area 

- Recommend kinetic destruction target 


2. Percentage of Codes 

The measurement model for macrocognitive research includes 18 macrocognitive 
process definitions which facilitate the categorization and measurement of 
macrocognition demonstrated in teams. Analysis of the Air Operations Center (AOC) 
dynamic targeting cell (DTC) communications revealed that during the TREX 09-1 
exercise, 13 out of 18 measurement model macrocognitive processes were used during 
the exercise. Additional non-measurement codes such as Administrative (ADMIN), 
Miscellaneous (MISC), Course of Action (COA) and Request Take Action (RTA) were 
assigned to cover gaps in the measurement model codes and our TREX 09-1 data. Table 
6 presents percentage of macrocognitive processes including administrative, 
miscellaneous, and Extra Code Eiller codes. 
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Table 6. 


Macrocognitive Process Code Percentages including Administrative, 
Miscellaneous and Extra Code Filler. 


Code 

Macrocognitive Process Categories 


Individual Knowledge Building 

Coder 1 # 

Coder 2 # 

Coder 1 % 

Coder 2 % 

IIG 

Individual Information Gathering 

537 

526 

17.00 

16.66 

IIS 

Individual Information Synthesis 

72 

33 

2.28 

1.04 

KOB 

Knowledge Object Development 

0 

0 

0.00 

0.00 

Team Knowledge Building 


TIE 

Team Information Exchange 

1192 

1187 

37.75 

37.59 

TKS 

Team Knowledge Sharing 

209 

172 

6.62 

5.45 

TSOG 

Team Solution Option Generation 

19 

11 

0.60 

0.35 

TENA 

Team Evaluation and Negotiation of Alternatives 

12 

4 

0.38 

0.13 

TPPR 

Team Process and Plan Regulation 

0 

0 

0.00 

0.00 

Internalized Team Knowledge 


ITK-TKS 

Team Knowledge Similarity 

2 

1 

0.06 

0.03 

ITK-TKR 

Team Knowledge Resources 

4 

2 

0.13 

0.06 

IK 

IK - Interpositional Knowledge 

3 

1 



ISA 

ISA - Individual Situational Awareness 

1 

1 



Externalized Team Knowledge 


ECSA 

Externalized Cue-Strategy Association 

12 

4 

0.38 

0.13 

PRTA 

Pattern Recognition and Trend Analysis 

4 

0 

0.13 

0.00 

UR 

Uncertainty Resolution 

1 

0 

0.03 

0.00 

Problem Solving Outcomes 


OOP 

Quality of Plan 

0 

0 

0.00 

0.00 

EPP 

Efficiency of Planning Process 

0 

0 

0.00 

0.00 

EPE 

Efficiency of Planning Execution 

0 

0 

0.00 

0.00 

Decision to Take Action 


COA 

Course of Action 

154 

149 

4.88 

4.72 

RTA 

Request Take Action 

81 

87 

2.56 

2.75 

Administrative, Miscellaneous, Statistical 


MISC 

Miscellaneous Actions/Comments 

666 

670 

21.09 

21.22 

ADMIN 

Administrative Actions/Comments 

185 

185 

5.86 

5.86 

ECF 

Extra Code Filler 

8 

127 

0.25 

4.02 


100.00 

100.00 


Table 6 presents the recalculated percentages using each coder’s individual code 
assignments and using Extra Code Filler codes which ensured both coders assigned the 
exact same number of codes. Table 7 percentages were calculated using only the coder’s 
individual code assignments. Coder 1 assigned 2,299 codes and coder 2 assigned 2,176 
codes. 
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Table 7. 


Macrocognitive Process Code Percentages excluding Administrative, 
Miscellaneous and Extra Code Filler. 


Code 

Macrocognitive Process Categories 


Individual Knowledge Building 

Coder 1 # 

Coder 2 # 

Coder 1 % 

Coder 

2% 

MG 

Individual Information Gathering 

537 

526 

23.36 

24.17 

IIS 

Individual Information Synthesis 

72 

33 

3.13 

1.52 

KOB 

Knowledge Object Development 

0 

0 

0.00 

0.00 

Team Knowledge Building 


TIE 

Team Information Exchange 

1192 

1187 

51.85 

54.55 

TKS 

Team Knowledge Sharing 

209 

172 

9.09 

7.90 

TSOG 

Team Solution Option Generation 

19 

11 

0.83 

0.51 

TENA 

Team Evaluation and Negotiation of 
Alternatives 

12 

4 

0.52 

0.18 

TPPR 

Team Process and Plan Regulation 

0 

0 

0.00 

0.00 

Internalized Team Knowledge 


ITK- 

TKS 

Team Knowledge Similarity 

2 

1 

0.09 

0.05 

ITK- 

TKR 

Team Knowledge Resources 

4 

2 

0.17 

0.09 

IK 

IK - Interpositional Knowledge 

3 

1 



ISA 

ISA - Individual Situational Awareness 

1 

1 



Externalized Team Knowledge 


ECSA 

Externalized Cue-Strategy Association 

12 

4 

0.52 

0.18 

PRTA 

Pattern Recognition and Trend Analysis 

4 

0 

0.17 

0.00 

UR 

Uncertainty Resolution 

1 

0 

0.04 

0.00 

Problem Solving Outcomes 


QOP 

Quality of Plan 

0 

0 

0.00 

0.00 

EPP 

Efficiency of Planning Process 

0 

0 

0.00 

0.00 

EPE 

Efficiency of Planning Execution 

0 

0 

0.00 

0.00 

Decision to Take Action 


COA 

Course of Action 

154 

149 

6.70 

6.85 

RTA 

Request Take Action 

81 

87 

3.52 

4.00 


100.00 

100.00 


Table 7 reflects overall code percentages after removing the extraneous codes 
administrative, miscellaneous and extra code filler codes. Administrative and 
miscellaneous communications were prevalent throughout the exercise but do not 
represent a cognitive process therefore that type of communications falls outside the 
scope of analysis for this thesis. The most frequently assigned codes, as shown in Table 
6 were Team Information Exchange, 37 percent, miscellaneous, 21 percent, and 
Individual Information Gathering, approximately 17 percent. Table 7 shows the 

59 
























































recalculated percentages of the macrocognitive processes used when the administrative, 
miscellaneous, and ECF codes were removed from the calculations. Team Information 
Exchange, 53 percent, and Individual Information Gathering, 24 percent, had the highest 
usage. After removing the extraneous codes, the Decision to Take Action (combination 
of Course of Action and Request Take Action codes) encompasses approximately 11 
percent of the communications and Team Knowledge Sharing makes up about 8 percent 
of coded communications. 

3. Code Trends 

One of our main assumptions before analyzing and coding the TREX 09-1 data 
was that there would be a need for a tremendous amount of information sharing amongst 
internal and external team members and a lot of information gathering by individual team 
members required to appropriately fix, find, track, target, engage and asses dynamic 
targets. After removing the extraneous codes assigned to the dynamic targeting center 
data, the percentages (shown in Table 7) indicate that approximately 53 percent of the 
data was coded as Team Information Exchange and approximately 24 percent was coded 
as Individual Information Gathering. Roughly 77 percent of the coded data is attributed 
to information gathering or information relay which validated our main assumption of 
large amounts of information sharing and passing. 

Another assumption that we had regarding the data is that more information 
gathering would take place earlier in the exercise in order for individuals to build an 
initial mental picture and become familiar with exercise and associated scenario 
parameters. Table 8 shows the number of Individual Information Gathering codes 
assigned across the four day exercise. Contrary to our original assumption, exercise day 
4 produced the most Individual Information Gathering codes of approximately 160 
compared to a previous high of roughly 135 codes that was assigned to exercise day 2 
data. 
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Table 8. 


Individual Information Gathering Day 1-4 Totals. 


Individual Information Gathering 


Coder 1 

Coder 2 

Day #1 

107 

103 

Day #2 

139 

133 

Day #3 

131 

131 

Day #4 

160 

159 

Total 

537 

526 


After studying the TREX 09-1 indoctrination and exercise preparation guide and 
further analyzing the data on exercise day 4, we attributed the higher coding percentage 
to be a result of multiple dynamic scenarios being run on exercise day 4. This resulted in 
the participants needing to clarify and gather more information than on the previous three 
days. Additionally, the TREX 09-1 exercise planners did a solid job on preparing the 
exercise participants by conducting an INCHOP brief and reviewing the TREX 09-1 
indoctrination and exercise preparation guide with all participants prior to the start of the 
exercise. This resulted in less need for the participants to share and gather initial 
information. Table 9 shows Team Information Exchange from day I compared against 
day 4 which also supports this claim. 

Table 9. Team Information Exchange Day I and Day 4 Totals. 


Team Information Exchange 


Coder 1 

Coder 2 

Day #1 

225 

235 

Day #4 

313 

317 


Another assumption was that there would be more Decisions to Take Action 
codes (combination Course of Action and Request Take Action codes) occurring later in 
the TREX 09-1 exercise. This initial assumption was based on the way most typical 
exercises are developed to build upon a climatic end point requiring several decisions to 
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be made typically near the end of the exercise. Table 10 lists the number of Course of 
Action and Request Take Actions codes assigned throughout the four day exercise. 

Table 10. Course of Action and Request Take Action Day 1 through Day 4 Totals. 


Course of Action 


Coder 1 

Coder 2 

Day #1 

23 

20 

Day #2 

42 

43 

Day #3 

42 

42 

Day #4 

47 

44 

Total 

154 

149 


Request Take Action 


Coder 1 

Coder 2 

Day #1 

15 

13 

Day #2 

27 

34 

Day #3 

22 

23 

Day #4 

17 

17 

Total 

81 

87 


The code numbers indicate that scenarios on day one and day four required the 
least decision making and that day 2 and day 3 consistently required approximately the 
same amount of Course of Action and Request Take Action. 

4. New Codes and Modifying Definitions 

During the practice coding process it was determined by both coders that the new 
set of macrocognitive process definitions included in the model failed to appropriately 
address and specifically define all the cognitive processes for the data that was analyzed. 
Specifically, there were no macrocognitive process definitions under the new set of 
definitions that appropriately classified or defined exercise personnel’s decision to take 
action. Decisions to Take Action were classified into two sub-categories, Course of 
Action and Request Take Action. Course of Action was assigned to a speech turn that 
issued an order for a more significant action that would be more likely to affect the 
overall scenario outcome. An order usually was issued from a more key position or 
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senior person down to a lower level position or subordinate. Request Take Action was 
assigned to speech turns that were lower-level requests between peers to take some 
action, but would not likely affect the entire outcome of the scenario. Other codes that 
were assigned but not part of the set of macrocognitive process definitions are 
miscellaneous and administrative. Miscellaneous codes were assigned to speech turns 
that did not include a macrocognitive process but were part of normal closed-loop 
communications such as “Roger”. Administrative codes were assigned to speech turns 
that were necessary for exercise support but not relevant or pertinent to the exercise 
scenarios such as communications checks prior to start of the exercise. 

B. INTER-RATER RELIABILITY 

Table 11 is a pivot table that compares coder 1 codes against coder 2 codes. Coder 
1 codes are read down and coder 2 codes are read across. Coder matches run diagonally 
through the pivot table starting with code Administrative 183, Miscellaneous 663, Team 
Information Exchange 1134, Individual Information Gathering 521, Individual 
Information Synthesis 23, Externalized Cue-Strategy Association 4, Team Knowledge 
Sharing 136, Course of Action 138, Request Take Action 78, Team Solution Option 
Generation 6, Team Evaluation and Negotiation of Alternatives 3, Internalized Team 
Knowledge 3, Uncertainty Resolution 0, Pattern Recognition and Trend Analysis 0, and 
Extra Code Eiller 0. 
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Table 11. 


Coder Pivot Table 


CODE 

TITLE 

ADMIN 

OSIIAI 

TIE 


w 

ECSA 

TKS 

COA 

< 

1 — 
cc 

TSOG 

TENA 

is: 

H 

cc 

3 

< 

1 — 
cc 

Cl 

ECF 

Total Coder 2 

ADMIN 

183 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

1 

185 

MISC 

0 

663 

6 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

670 

TIE 

0 

1 

1134 

5 

9 

1 

24 

9 

0 

2 

0 

0 

0 

1 

1 

1187 

IIG 

2 

0 

2 

521 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

526 

IIS 

0 

0 

4 

0 

23 

0 

6 

0 

0 

0 

0 

0 

0 

0 

0 

33 

ECSA 

0 

0 

0 

0 

0 

4 

0 

0 

0 

0 

0 

0 

0 

0 

0 

4 

TKS 

0 

0 

17 

1 

6 

1 

136 

3 

0 

3 

3 

1 

0 

1 

0 

172 

COA 

0 

0 

4 

1 

0 

0 

1 

138 

1 

1 

0 

0 

0 

0 

3 

149 

RTA 

0 

0 

4 

1 

0 

0 

1 

0 

78 

0 

0 

0 

0 

0 

3 

87 

TSOG 

0 

0 

1 

0 

1 

2 

0 

0 

0 

6 

1 

0 

0 

0 

0 

11 

TENA 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

3 

0 

0 

0 

0 

4 

ITK 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

3 

0 

0 

0 

3 

UR 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

PRTA 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

ECF 

0 

2 

20 

8 

33 

4 

39 

3 

2 

6 

5 

2 

1 

2 

0 

127 

Total 
Coder 1 

185 

666 

1192 

537 

72 

12 

209 

154 

81 

19 

12 

6 

1 

4 

8 

3158 


The pivot table also illustrates what eodes the eoders disagreed upon. Under the 
Administrative (ADMIN) eategory, eoder 1 and eoder 2 both had assigned a total of 185 
administrative eodes to the data. However, both eoders only matehed seleetions for 183 
out 185 administrative eodes. There were two disagreements per eaeh eoder. Reading 
down the eolumn under ADMIN you ean see that eoder 2 had seleeted two Individual 
Information Gathering (IIG) eodes when coder I had assigned administrative codes. 
Additionally, reading across the row for category ADMIN, you can see that coder I 
selected 1 Team Knowledge Similarity (TKS) and 1 Extra Code Filler (ECF) code when 
coder 2 had assigned an ADMIN code. Reasons for disagreement between coders can be 
attributed to a particular TREX 09-1 line of text not being interpreted the same way 
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between the coders (i.e., coder 1 thought the line of text related to the actual exercise 
whereas coder 2 classified it as non-essential communications assigning an administrative 
code). 

One thing that stands out after going through each code category above and 
comparing the common differences between coders (i.e., when one coder chose X the 
other coder chose Y) is that when there was a disagreement on Team Information 
Exchange (TIE) or Team Knowledge Sharing (TKS) codes, the other code typically 
selected code was Team Knowledge Sharing and Team Information Exchange 
(respectively). Reading down the pivot table for category TIE, you can see that the 
number of agreed Team Information Exchange codes is 1,134 and that coder 2 disagreed 
with coder I and selected Team Knowledge Similarity 17 times. Additionally, reading 
down Team Knowledge Sharing (TKS) category, both coders agreed 136 times but coder 
2 disagreed and selected Team Information Exchange 24 times. The disagreement 
between TKS and TIE and the patterned alternative response of the other coder code 
indicates that there is some ambiguity in the measurement model definition for both 
codes. Eurthermore, when coder 1 selected Team Information Exchanges and Team 
Knowledge Sharing, coder 2 disagreed and selected the Extra Code Eiller (ECE) code 20 
and 39 times (respectively). This disagreement between coders and non selection of the 
measurement model code indicates that the definitions for Team Knowledge Sharing and 
Team Information Exchange need to be modified to remove ambiguity and vagueness. 

1. Kappa Cohen Statistic Analysis 

Kappa was calculated using the equation listed in Eigure 22. Probability of 
Agreement, Pr(a), is calculated by adding up all the agreements of the fifteen codes used 
and dividing that number by the total codes assigned. Pr(a) = (183 -i- 663 -i- 1134 -i- 521 -i- 
23 -t 4 -I- 136 -t 138 -t 78 -t 6 -t 3 -t 3 -t 0 -t 0 -t 0) / 3158 = .915769474. Probability of 
Agreement Due to Randomness, Pr(e), is calculated by multiplying coder 1 categorical 
codes against coder 2 categorical codes (i.e., ADMIN * ADMIN, MISC * MISC, etc..), 
summing the total and dividing that number by the total codes times the total codes. 
Pr(e) = (185 * 185 -t 666 * 670 -t 1192 * 1187 -i- 537 * 526 -t 72 * 33 -t 12 * 4 -i- 209 * 
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172 + 154 * 149 + 81 * 87 + 19 * 11 + 12 * 4 + 6 * 3 + 1 * 0 + 4 * 0 + 8 * 127) / (3158 


* 3158) = .225355972. Figure 12 shows the calculation of the Kappa Cohen equation 
using our statistic numbers. 




Pr{a)-Pr(e) . 915769474-.225355972 


1-Pr(^) 


1-.225355972 


=. 891265507^.89 


Figure 21. Kappa Cohen Statistical Equation Result. 


Our .89 returned Kappa value indicates that coder 1 and coder 2 both interpreted 
the measurement model definitions and TREX 09-1 exercise data the same way. A 
returned kappa (k) value ranging between .81 - 1 is considered an almost perfect 
agreement between coders (Eandis & Koch, 1977). 

C. COGNITIVE PHASES 


I. First Stage Macrocognitive Phase 

Prior to being assigned to an AOC, service members are often interviewed and 
have their professional records screened to ensure the member has the adequate formal 
schooling, training and experience level to successfully perform AOC tasks. Based on 
the type of training and level of experience, team members assigned to an actual AOC or 
just participating in a coordinated exercise such as a TREX, bring different levels of 
knowledge that can be applied to various dynamic situations. In the first stage of the 
measurement model, this existing knowledge is referred to as Internalized Team 
Knowledge. According to the measurement model. Internalized Team Knowledge is the 
knowledge held in the mind of an individual team member. Our coding of the TREX-09 
communications data revealed that only .15 percent of the total data coded was 
Internalized Team Knowledge. Internalized Team Knowledge consists of two cognitive 
process subcategories called Team Knowledge Similarity and Team Knowledge 
Resources. 
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Team Knowledge Similarity looks at how well team members in different jobs 
within the AOC understand each other’s roles and responsibilities. A generic example of 
Team Knowledge Similarity would be how well does the intelligence officer understand 
the role of the JAG officer. Team Knowledge Resources focuses on the team’s overall 
collective understanding of the task (SUMMIT Measurement Model Document). Coded 
TREX-09 communication data did not provide any definitive reasons that explained the 
team member’s low usage of the Internalized Team Knowledge cognitive process. The 
specific reason for the low usage may lie in further explanation of the Internalized Team 
Knowledge definition. According to the Salas et al, raw data associated with internalized 
knowledge should be collected prior to the start of the exercise on all team members and 
then an after exercise assessment survey should be given to see how well they understood 
their responsibility and the overall task. 

It is not clear whether the above listed pre- and post-assessments were conducted, 
and no such data has been collected or sent to the coders. Without the pre- and post¬ 
assessment data, it is not possible for us to ascertain team members true Internalized 
Team Knowledge and understanding of roles and tasks by solely looking at the TREX 
09-1 coded data. It is not fair to say that the low score of Internalized Team Knowledge 
cognitive processes coded by us is a true and accurate representative of TREX-09 team 
members overall internali z e knowledge. However, evaluation of previous iterations of 
the SUMMIT model using Northeast Air Defense Sector and Eederal Aviation 
Administration communications data from September 11, 2001 by Euis E. Socias also 
reflected similar low Internalized Team Knowledge usage and coded scores. 

2. Second Stage Macrocognition Phase 

Individual Knowledge Building, the second cognitive process in the measurement 
model, focuses on the actions taken by team members to increase their overall knowledge 
of a given situation. Even with years of training and schools, individuals may find 
themselves in situations that require other steps be taken to build on their existing 
knowledge. According to the SUMMIT definitions, such steps could involve but are not 
limited to reading and asking questions. Table 12 includes excerpts from TREX-09 
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showing team members engaging in Individual Information Gathering (IIG) and 
Individual Information Synthesis (IIS). This ineludes asking for elarifieation on 
information previously passed sueh as the name of a missing airfield to requesting 
speeifie data on a known bomb maker. 


Table 12. Individual Knowledge Building: Individual Information Gathering and 
Individual Information Synthesis Examples. 


Individual Knowledge Building 

Originator 

Communication 

Code 

C2DO 

SIDO: I missed the name of the airfield? 

IIG 

C2DO 

SIDO: Type of aircraft we are looking for and its latitude and 

longitude? 

IIG 

SIDO 

Airfield is located at (*Removed*); type of aircraft is STOL cargo 

plane. 

TIE/TIE 

SODO 

Command control duty officer (C2DO), what is the capability to track 

STOL Cargo aircraft with (Tac C2)? 

IIG 

SIDO 

C2DO, STOL cargo aircraft has departed to the target; may operate 

between 44 and 200 knots. 

IIS 


Individual Knowledge Building 

Originator 

Communication 

Code 

lOT 

NTI: What can you tell me about bomb supplier #1? 

IIG 

NTI 

Information operations targeteer, bomb supplier #1 is a known materials 

supplier; communication on cover and will report any new intelligence 

when available. 

IIS/TIE 


The pereentage of TREX 09-1 team member speeeh turns eoded as Individual 
Knowledge Building was 16.66 pereent. Individual Knowledge Building was the seeond 
most used maeroeognitive process. The most used cognitive process was Team 
Knowledge Building. Although there is no definitive answer, the high usage of 
Individual Knowledge Building may have resulted from the unfamiliar dynamic setting 
of the exercise. The TREX-09 exercise placed team members in a time-compressed 

68 

















situation while at the same time requiring processing of a large amount of information 
from various intelligence sensors. To build on their existing knowledge and to maintain 
continuous situation awareness in the dynamic environment, team members engaged in 
building their individual knowledge by asking lots of questions. High Individual 
Knowledge Building speech turns were also recording in the Federal Aviation 
Administration and Fire Department of New York communication data during the 11 
September 2001 attack. This was probably attributed to the unfamiliar and changing 
environment factors. 

3. Third Stage Macrocognition Phase 

The third cognitive phase of the measurement model is Team Knowledge 
Building. Team Knowledge Building includes actions taken by team members to 
disseminate and transform information into actionable knowledge. In a dynamic 
environment, to be effective against time-sensitive targeting, AOC team members must 
be ready at all times to make tough decisions in a time limited environment. Speech 
turns during TREX-09 shows that team members were highly engaged in information 
exchange and sharing. In Table 13, team members discussed the effects of radiological 
fallout from a possible strike against a building. Such collaboration among different 
dynamic cells could ensure higher situation awareness and facilitates better informed 
decision making. 

Table 13. Team Knowledge Building: Team Information Exchange Example. 


Team Knowledge Building Process 

Originator 

Communication 

Code 

DEC 

Awaiting radiological impact assessment on watershed if the building is to be 

strike. Second, option in work is to destroy local roads to prevent access in/out. 

TIE/TSOG 

DECD 

Aircraft returns watershed non-issue 

TIE 

JOC_JCE 

Dynamic effect cell, you have high-value target on your dynamic target list. 

What is the air combat commander game plan? If you have a good one, I will 

appoint you the lead but I think SOE needs to be considered. 

TIE/IIG 
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Forty-three pereent of the TREX 09-1 team eommunieations were eoded as one of 
the maeroeognitive proeesses that oeeur during the Team Knowledge Building eognitive 
proeessing phase. Maeroeognitive proeesses that were used among the team inelude 
Team Information Exehange, Team Knowledge Sharing, Team Solution Option 
Generation, and Team Evaluation and Negotiation of Alternatives. Most frequently used 
maeroeognitive proeesses used were Team Information Exchange (37.57%) and Team 
Knowledge Sharing (5.45%). Other Team Knowledge Building maeroeognitive 
processes used during the exercise fell below 1 percent. The high usage of Team 
Information Exchange was probably due to the dynamic nature of the exercise and the 
complexity involved in engaging time-sensitive targets. Table 14 is an excerpt from the 
TREX-09 communications data that shows team members sharing information on rules 
of engagement and discussing the effects of a strike mission against an airfield. 

Table 14. Team Knowledge Sharing Example. 


Team Knowledge Building Process 

Originator 

Communication 

Code 

DEC 

Self defense applies for hostile acts from Country #3 fighters in Country 

# 2 or #4 airspace 

TKS 

DEC 

Enemy forces that employ ordnance, electronic attack or achieve a radar 

lock against friendly forces have committed a hostile act 

TKS 


TDO 

If we crater the runway and taxiways, we may be able to effectively stop 

the target. 

TSOG 

lOT 

Target Duty Officer (TDO): Just throwing this out there, but if you target 

the roadways, is there a chance you could spook them and they might fire 

off their missiles and run? 

TENA 


4. Fourth Stage Macrocognition Phase 

Externalized Team Knowledge refers to knowledge that has been agreed upon by 
members of the team and is the fourth maeroeognitive phase in the measurement model. 
Under the Externalized Team Knowledge phase, TREX-09 team members used all three 
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macrocognitive processes: Externalized Cue-Strategy Assoeiation, Pattern Reeognition 
and Trend Analysis, and Uneertainty Resolution. The pereentage of speeeh turns eoded 
as one of the maeroeognitive proeesses that oeeur in the Externalized Team Knowledge 
phase was only.24 pereent with ECSA and PRTA being the most used out of the fourth 
stage. Table 15 contains excerpts from TREX-09 showing team members eoming to an 
agreement on the type of weapon to use and the de-eonfliction needed. 


Table 15. Externalized Cue Strategy Assoeiation and Pattern Recognition Trend 

Analysis Examples. 


Externalized Team Knowledge 

Originator 

Communication 

Code 

DECSOLE 

The dynamic effect cell chief stated that if there is an erect launcher in a joint 

special operations area (JSOA) his rules of engagement (ROE) are to kill it as 

soon as possible and if there is time to de-conflict with the teams. 

ECSA 

JSOET 

Correct, if per joint force commander (JEC) 

TIE 

DECSOLE 

I can't remember ROE in the west for OIF but I think it was something similar. 

TIE 

DECSOLE 

He mentioned tomahawk land attack missiles (TEAM) wouldn't be de-conflicted 

either, but I dispute that logic. First, we wouldn't use a TEAM to shot a 

launcher.. .1 don't think. Unless it was a last resort. 

ECSA 

SECSOLE 

Second the flight time is great enough to pass the warning and do the de- 

confliction. 

TIE 

JSOET 

TLAMs most definitely have to be de-conflicted even for over flight of the JSOA, 

unless directed otherwise by the JEC. He's not the JEC. If any issues, let me 

know and I'll pass up to the joint special operation task force commander for 

discussion with the JEC. 

UR/ 

ECSA 


DEC 

looks similar to our first target with regards to unknown presence of Radiological 

containers in facility. We would look at interdiction for containment to prevent 

travel to/fm that site, your thoughts on best plan/option 

PRTA 


There is no definitive answer to explain why Externalized Team Knowledge 


communieations, the proeess where the team validates information for aeeuraey and 
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completeness before taking action, came in with such low percentages. Our attempt to 
explain the low Externalized Team Knowledge percentage is that most AOC personnel 
have years of experience and are trained to do their job with little or no assistance. 
Individual cells and operators communicated via chat with other team members after 
most of the analyses and final agreement on information received was already decided 
upon or completed. Due to the layout of the TREX 09-1, exercise personnel located in 
one room with divider walls between each position, it is highly probable that team 
members communicated with each other via voice vice utilizing chat only, which could 
have resulted in loss of possible Externalized Team Knowledge speech turns. According 
to the measurement model definitions, a voice recorder should be used to capture the 
exchange of information between teammates. 

5. Fifth Stage Macrocognition Phase 

The final macrocognitive phase in the measurement model is Team Problem 
Solving Outcomes. During the coding of the TREX 09-1 data there was no Team 
Problem Solving Outcomes cognitive processes found or coded. Team Problem Solving 
Outcomes focuses on the quality and speed by which team members come up with viable 
solutions to problems or develop response plans. Our possible explanation of why there 
were no Team Problem Solving Outcome speech turns is that the team members were 
dealing with unique dynamic situations which made it difficult to pull from past 
experiences and possibly which could have made exercise participants feel uncomfortable 
to suggest and/or recommend solutions. Additionally, Team Problem Solving Outcomes 
could have been communicated via voice to team members located in same room vice 
being sent via chat. 
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VII. CONCLUSION AND RECOMMENDATIONS 


A. CONCLUSIONS 

1. Use of Codes 

The use of TREX 09-1 communication data to evaluate the measurement model 
shows that throughout the exercise team members used 13 of the 18 macrocognitive 
process codes. Codes not used fall within three of the macrocognitive stages of the 
measurement model. Those not coded include quality of plan, efficiency of planning 
process, efficiency of planning execution, knowledge object development, and team 
process and plan regulation. The fact that they were not coded by us does not mean that 
they did not take place within the exercise. What it means is that the manner by which 
theses processes are captured was not possible when analyzing the TREX 09-1 
communication chat room data. In other words, these codes not used by us require data 
to be recorded, time stamp, or be shown graphically so as to measure the outcome and 
object development of the team. 

2. Code Percentage and Kappa Cohen Results 

The code percentage results and the Kappa Cohen analysis assisted in empirically 
evaluating the SUMMIT measurement model for macrocognitive research. The use of 13 
of the measurement model codes is evident throughout the four analyzed exercise 
sessions. The high, almost perfect. Kappa Cohen result further indicates that both coders 
had a clear interpretation of the measurement model code definitions and assigned the 
codes consistently throughout the coding process. 

B. RECOMMENDATIONS 

We recommend trying to locate or collect other DTC data from TREX exercises 
in addition to re-analyzing past maritime interdiction operation and 9/11 Eire Department 
data using the measurement model for macrocognitive research. Analyzing and coding 
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other data sets using the same measurement model for macrocognitive research will allow 
side by side comparisons of number of macrocognitive codes used in team collaboration. 
The comparative analysis can aid in identifying gaps between the measurement model 
code definitions and the macrocognitive processes of individuals and teams during real- 
world scenarios, compared to laboratory experiments. Data could support modifications 
to the measurement model code definitions or lead to new codes being added (i.e., 
Request Take Action, Course of Action). 

Recommend taking past and future DTC and maritime interdiction operation data 
and re-calculate/calculate inter-rater reliability between coders using Kappa Cohen. The 
data generated from the Kappa Cohen process (i.e., pivot chart indication of non 
agreement between coder assigned codes) is extremely useful in determining if code 
definitions are too vague. Adjustment of code definitions and recoding and calculating 
inter-rater reliability can be a metric for testing definition ambiguity and if adjustment 
aided or worsened the agreement level. 

Recommend that pre-exercise coordination be achieved with the Armed Forces 
Research Laboratory War-fighter Readiness Research Division prior to the next TREX 
exercise event. Pre-coordination could allow for the use of better macrocognition 
measurement tools and techniques that would be instrumental in capturing more 
macrocognitive processes information. Devices such as voice recorders could be used on 
the main exercise floor to capture exercise personnel speech turns that were spoken due 
to proximity of other exercise personnel vice being sent as text entries through Microsoft 
internet relay chat communications system. Pre-survey and post-surveys could also be 
administered to exercise personnel that would help in determining if new knowledge was 
actually generated and/or produced during Individual Knowledge Building and Team 
Knowledge Building phases. Use of eye tracking equipment could be incorporated at 
every exercise participant workstation to track their eye movement leading to information 
collection and possible development of better command and control graphical user 
interface systems. 
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